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FOREWORD 


This  report  describes  a  study  performed  by  the  Federal  Sys- 
terns  Division,  IBM  Corporation,  7220  Wiscc  sin  Avenue,  Bethes- 
da,  Maryland,  under  Contract  AF  30(602)-3563  to  the  Rome  Air 
Development  Center,  Air  Force  Systems  Command,  Griffiss  AFB, 
New  York..  The  study,  performed  during  the  period  November 
23,  1964  to  November  22,  1965,  considered  the  principal  para¬ 
meters  affecting  content  analysis  of  documents  by  a  technique  based 
upon  multiple  discriminant  functions. 
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performing  the  experiments;  and  to  Mr.  Robert  Phillips  of  the  IBM 
Corporation  for  his  consultation  in  devising  the  hypotheses  and  ex¬ 
periments. 
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ABSTRACT 


A  series  of  experiments  was  performed  to  investigate  the 
effectiveness  and  utility  of  automatically  classifying  documents 
through  the  use  of  multiple  discriminant  functions.  Classification 
is  accomplished  by  computing  the  distance  from  the  mean  vector 
of  each  category  to  the  vector  of  observed  frequencies  of  a  docu¬ 
ment  and  assigning  the  document  to  the  category  having  the  high¬ 
est  probability.  Data  concerning  the  effect  of  the  principal 
classification  parameters  on  classification  performance  is  re¬ 
ported,  based  on  a  data  base  of  approximately  2700  abstracts  from 
the  solid  state  physics  field.  The  parameters  studied  were  the 
number  of  sample  documents  required  to  define  a  category,  the 
length  of  documents,  the  interrelationship  of  the  number  of  sample 
documents  and  their  lengths,  the  relation  of  the  number  of  word 
types  in  a  document  to  the  number  of  categories  assigned  to  it, 
levels  in  a  structure,  homogeneity  of  categories,  and  performance 
measures.  A  higher  performance  level  was  obtained  when  samples 
of  140  documents  were  used  to  define  each  category  than  with  sam¬ 
ples  of  35  anu  70  documents.  Classification  results  obtained  on  in¬ 
dependent  t^t  sets  of  documents  ranged  from  73  to  92  percent.  The 
test  sets  contained  419  and  1333  documents.  Results  are  also  re¬ 
ported  in  terms  of  Swets'  effectiveness  measure  and  Cleverdon’s 
ratios  of  relevance,  recall  and  precision. 


EVALUATION 


The  purpose  of  this  work  was  to  identify  parameters  which 
affect  the  accuracy  of  an  automatic  document  classification 
technique  based  on  the  use  of  multiple  discriminant  functions. 
This  study  was  necessary  prior  to  testing  the  technique  in  an 
operational  environment. 

The  most  significant  parameter  found  was  the  number  of 
sample  documents  used  to  define  each  category.  By  increasing 
the  number  of  sample  documents,  improvements  were  realized 
in  both  accuracy  and  the  set  of  discriminating  word  types 
selected  to  represent  each  category.  In  addition,  document 
length  was  found  to  have  no  adverse  affect  on  accuracy. 

Arrival  rate  of  new  word  types  approached  a  reasonably  sma]l 
number  and  accuracy  improved  at  lower  levels  in  the  classi¬ 
fication  structure. 

As  a  result  of  this  effort,  this  technique  will  be  tested 
in  a  user  environment  where  improvements  from  user  suggestions 
and  reactions  should  result  in  an  operational  classification 
technique. 
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Section  I 


INTRODUCTION 


With  the  increasing  use  of  optical  character  recognition  equipment,  di¬ 
rect  teletype  input  and  other  forms  of  high-speed,  high-volume  input  devices 
for  computers,  the  processing  of  narrative  documents  is  becoming  more  prac¬ 
tical.  Methods  are  required  for  screening,  writing,  and  disseminating  such 
documents;  for  indexing  and  abstracting  them;  and  for  extracting  important 
information  from  them.  While  techniques  for  performing  such  functions  auto¬ 
matically  on  formatted  or  semi-formatted  documents  have  been  in  use  for  some 
time,  the  low  density  of  information  and  wide  variety  of  ways  of  expressing  it 
have  made  the  development  of  methods  for  processing  narrative  text  more 
difficult. 

This  stuo>  was  undertaken  as  part  of  a  continuing  effort  to  develop  a  use¬ 
ful  and  reliable  technique  for  automatic  content  analyst  of  documents  con¬ 
taining  primarily  narrative  text.  Content  analysis  is  a  key  element  of  the 
screening,  routing,  disseminating,  classifying  and  indexing  functions.  In 
general,  the  present  work  is  directed  toward  automatically  assigning  a  label 
or  labels  from  a  predetermined,  finite  set  of  possible  labels  to  a  given  sample 
of  text.  The  particular  application  can  be,  for  example,  assigning  index  terms, 
choosing  nodes  in  a  classification  structui  e,  determining  whether  a  document 
should  be  sent  to  a  certain  analyst. 

Several  investigators  have  proposed  techniques  for  these  purposes  using 
a  variety  of  approaches.  One  particularly  effective  approach  is  the  application 
of  the  statistical  technique  of  multiple  discriminant  functions.  This  technique 
was  tirst  investigated  in  IBM's  1RAD  Task  #0274  and  is  described  in  the  task 
final  report^).  A  suosequent  series  of  experiments  was  reported  in  a  techni¬ 
cal  paperl-).  Following  two  more  successful  sets  of  experiments,  the  present 
study  was  undertaken  to  determine  the  principal  parameters  affecting  content 
analysis  of  documents  as  performed  by  the  above  technique.  The  technique 
was  held  constant  while  the  effects  of  the  various  parameters  involved  in  its 
use  were  studied.  The  data  base  on  which  the  experiments  were  made  con¬ 
sisted  of  a  set  of  approximately  2700  abstracts  of  documents  in  the  field  of 
solid  state  physics.  The  results  of  the  study  lead  both  to  effective  application 
of  the  technique  and  to  suggested  ways  of  improving  its  performance. 
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Section  II 


OBJECTIVES  OF  THE  CONTRACT  EFFORT 


The  objectives  of  this  contract  effort  Here  to  determine  the  principal 
parameters  affecting  classification  of  documents  by  an  automatic  technique 
employing  multiple  discriminant  functions,  and  to  investigate  their  effect  on 
the'  utility  and  effectiveness  of  the  classification  technique.  The  parameters 
to  be  studied  were: 

a.  Number  of  sample  documents  used  to  define  a  category 

I).  Length  of  documents  measured  in  terms  of  the  number  of  word  tokens 
and  the  number  of  sentences  in  it 

c.  Interrelationship  between  the  number  of  documents  and  their  lengths 
used  to  define  a  category 

d.  Relation  of  the  number  of  word  types  in  a  document  to  the  number  of 
concepts  (categories)  assigned  to  that  document 

e.  Relation  between  the  level  in  a  structure  and  classific  ation  perfor¬ 
mance 

f.  Homogeneity  of  a  category 

g.  Metric  used  to  measure  the  effect  of  the  parameters  on  classification 
performance. 

The  technique  requires  a  sample  of  documents  known  to  belong  to  each 
category  to  compute  coefficients  for  the  classification  decision.  As  the  num¬ 
ber  of  sample  documents  is  increased,  a  better  estimate  of  the  discriminant 
coefficients  can  be  expected.  However,  as  the  number  of  sample  documents 
increases  the  number  of  different  word  types  observed  also  increases.  The  two 
problems  are  to  find  a  subset  of  types  and  a  reasonable  number  of  documents 
from  which  reliable  estimates  of  means  and  dispersion  can  be  computed.  A  sam¬ 
ple  is  required  for  each  category  so  that  ;f  a  large  number  ol  documents  were 
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required  for  reliable  estimates,  the  technique  might  become  uneconomical. 
One  would  expect  that  fewer  long  documents  are  required  to  obtain  reliable 
estimates;  hut  since  long  documents  arc  apt  to  add  more  new  types  to  the 
word  population,  the  need  to  study  the  interrelationship  of  the  length  of  docu¬ 
ments  cxistc  I. 

Overall  classification  performance  may  be  impaired  by  the  nature  of 
certain  categories.  These  categories  may  be  less  homogeneous  than  others. 
Lack  of  homogeneity  in  a  category  may  be  due  to  the  definition  cf  its  subject, 
the  sample  documents  representing  it,  or  the  subset  of  words  selected  to 
represent  it.  A  method  of  recognizing  lack  of  homogeneity  and  its  effect  on 
classification  performance  needs  to  he  determined. 

Some  classification  techniques  perform  well  when  distinguishing  between 
general  subjects,  but  need  further  study  to  assess  their  performance  at  more 
detailed  subject  levels.  Rather  than  classify  documents  on  broad  subjects 
such  as  chemistry,  physics  and  biology,  experiments  were  conducted  at  de¬ 
tailed  levels  within  the  field  of  solid  state  devices,  distinguishing  between  the 
application,  metallurgical,  chemieal  and  physical  properties  cf  devices. 
Further  experiments  were  performed  to  distinguish  between  specific  magnetic 
properties. 

In  operational  situations  many  documents  are  classified  into  more  than 
one  category.  A  classification  technique  must  also  be  able  to  adequately 
classify  this  type  of  document.  Investigations  are  required  to  dctci  nine  if 
this  class  of  documents  contains  statistical  properties  different  from  docu¬ 
ments  belonging  to  only  one  category. 

Since  some  of  the  parameters  are  interrelated,  their  individual  effect 
in  I  hr  multivariate  model  cannot  be  measured  easily.  A  metric  is  needed  to 
measure  the  contribution  of  each  parameter  to  classification  performance. 

As  in  all  parameter  studies  the  metric  must  be  invariant  with  respect  to 
the  data  set,  otherwise  comparisons  between  various  applications  would  be 
impossible. 

The  objective  of  this  set  of  tasks  was  to  provide  an  understanding  of  the 
capabilities  and  limitations  of  a  statistical  classification  technique.  Isolation, 
definition  and  measurement  of  classification  parameters  is  an  important  step 
that  must  be  accomplished  prior  to  prototype  testing  of  any  new  technique. 
Once  the  ranges  and  effects  of  the  classification  parameters  are  established, 
applicability  to  specific  systems,  environments,  or  needs  can  then  be  evalu¬ 
ated. 
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Section  III 


CLASSIFICATION  TECHNIQUE 


This  section  contains  an  operational  description  of  the  classification  pro¬ 
cedure,  a  brief  discussion  of  the  multiple  discriminant  functions  used,  and  a 
short  summary  of  the  computer  programs  used  to  implement  the  technique 
experimentally . 


OPERATIONAL  DESCRIPTION 

The  problems  encountered  in  a  classification  technique  are:  selection  of 
the  variables  on  which  the  classification  decision  is  to  be  based,  computation  of 
the  classification  statistics,  determination  of  the  decision  rule,  and  analysis  of 
effectiveness . 

A  user  starts  with  a  set  of  documents  and  decides  on  a  group  of  subjects 
that  interest  him.  He  then  constructs  a  tree  showing  the  classification  structure 
desired  for  the  total  collection  of  documents  under  consideration.  Each  node  of 
the  tree  represents  a  category,  and  all  nodes  or  branches  emanating  from  that 
node  are  its  subcategories.  A  category  number  is  assigned  to  each  node.  Each 
document  from  a  sample  of  the  total  collection  is  manually  assigned  to  one  or 
more  nodes  of  the  tree. 

The  variables  used  in  the  classification  process  are  frequency  statistics  on 
word  types  from  each  category.  Previous  experiments  indicate  that  all  word 
types  do  not  need  to  be  retained  for  the  classification  process,  and  computation¬ 
ally  it  would  be  impractical  to  do  so.  Ideally,  words  selected  to  represent  tne 
categories  should  occur  in  one  and  only  one  category.  However,  there  are  usu¬ 
ally  only  a  few  words  in  any  data  base  that  occur  in  one  and  only  one  category  , 
and  these  words  do  not  necessarily  occur  in  every  document.  Therefore,  a 
statistic  is  needed  to  identify  words  approximating  this  condition.  Such  words 
should  have  a  small  within -category  variance  and  a  large  among -cate  gory 
variance;  i.e.,  their  frequency  should  be  close  to  the  mean  for  one  category 
but  far  from  the  mean  for  all  other  categories.  The  F  ratio  of  among -category 
category  variance  and  pooled  within -category  variance  meets  this  condition  and 
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has  been  used  as  a  word  selection  statistic.  The  F  ratio  is  also  similar  to  the 
multivariate  maximizing  condition  of  discriminant  analysis;  therefore,  words 
having  a  high  value  of  F  should  also  have  high  multiple  discriminant  coefficients. 

An  F  ratio  is  computed  for  each  word  type  after  its  frequency  of  occurrence 
in  each  sample  document  is  counted  and  the  mean  frequencies,  pooled  within  - 
category  variances  and  arr.ong-category  variances  are  calculated.  A  subset  of 
words,  called  the  discriminating  word  set,  is  selected  for  retention  for  compu¬ 
tation  of  more  precise  weighting  coefficients  for  use  in  the  classification  equa¬ 
tion.  Words  selected  for  retention  must  not  only  have  a  high  F  ratio  but  must 
also  have  a  high  likelihood  of  occurring  in  many  documents.  (Otherwise,  too 
many  words  would  have  to  be  retained  to  achieve  an  adequate  level  of  coverage 
in  each  document.) 

Finally,  to  represent  each  category  in  this  experiment,  s  words  are  se¬ 
lected,  where  s  =  50/q;  q  is  the  number  of  categories,  and  s  is  an  integer 
formed  by  truncating  the  remainder  of  the  quotient.  The  words  in  descending 
order  of  F  are  examined  and  for  each  category  the  first  s  words  having  a  mean 
greater  than  0.2  are  selected.  When  this  procedure  is  completed,  n  =  qs  £  30 
words  will  have  beer  selected.  In  the  classification  experiments  16  words  were 
selected  for  each  of  the  three  categories.  (A  program  limitation  currently  pre¬ 
cludes  the  selection  of  more  than  50  types.) 

Having  defined  a  multidimensional  space  with  48  words,  the  sample  docu¬ 
ments  are  used  again  to  compute  means  and  dispersion  of  each  category  and 
jr  the  weighting  coefficients  of  each  word  for  the  multiple  discriminant  func¬ 
tions  .  One  set  of  multiple  discriminant  functions  is  computed  for  each  node 
(i.e.,  group  of  categories)  in  a  structure.  The  weighting  coefficients  for  each 
word  are  determined  by  the  context  in  which  it  is  used.  The  center  (centroid) 
of  each  category  is  represented  by  a  vector  of  mean  frequencies  derived  from 
sample  documents  known  to  belong  to  that  category. 

A  new  document  is  classified  by  counting  the  frequencies  of  the  words 
occurring  in  it  and  comparing  this  vector  of  observed  frequencies  with  the  mean 
vector  of  every  category.  The  probability  of  membership  in  each  category  is 
computed  and  the  document  is  assigned  to  the  category  having  the  highest  proba¬ 
bility.  For  applications  in  which  assignment  to  one  category  s  not  desirable, 
the  probabilities  may  be  stored  for  future  retrieval  requests .  These  proba¬ 
bilities  may  be  considered  relevance  values,  and  responses  to  queries  may  be 
ranked  in  descending  order  of  the  probabilities. 
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MULTIPLE  DISCRIMINANT  FUNCTIONS 


Many  statistical  techniques  exist  for  the  classification  of  a  random  observa¬ 
tion  into  one  of  two  populations.  However,  not  until  recently  have  techniques 
been  developed  for  classifying  observations  into  many  categories.  A  survey  of 
the  techniques  has  indicated  that  multiple  discriminant  functions  appear  to  be 
the  best  statistical  technique  for  document  classification.  They  are  functions 
that  contain  weighting  coefficients  which  indicate  the  discrimination  ability  of 
each  variable.  They  also  provide  an  advantage  over  linear  discriminant 
functions  since  pairwise  comparisons  between  two  categories  at  a  time  are  un¬ 
necessary.  The  relative  probability  of  membership  in  each  category  can  be 
obtained  with  a  single  statistic  for  each  category. 

Hodges^,  and  Tatsuoka  and  Tiedman^  have  provided  excellent  surveys 
of  classification  and  discrimination  techniques.  The  original  work  in  discrimi¬ 
nant  analysis  w’as  performed  by  Fisher  and  which  Barnard^  applied  to  a 
problem  dealing  with  the  classification  of  skulls  into  archeological  time  periods . 
From  a  representative  sample  of  each  skull,  the  average  value  of  each  measure¬ 
ment  w'as  computed  for  each  period.  A  new-  skull  could  then  be  classified  by 
comparing  differences  in  the  observed  value  with  the  expected  value  of  each  of 
the  seven  measurements.  Rather  than  sum  these  differences  with  equal  weight, 
Fisher  devised  a  technique  for  computing  a  coefficient  for  each  difference.  The 
desired  set  of  coefficients  would  be  the  one  that  gave  greater  weight  to  those 
measurements  which  provided  greater  discrimination  among  the  time  periods. 
This  set  of  coefficients  yields  a  linear  discriminant  function.  The  optimum  set 
of  coefficients  is  found  by  maximizing  the  ratio  of  the  among-category  sum-of- 
squares  of  this  function  to  its  pooled  within-categories  sum-of-squares. 

Further  research  has  been  performed  in  discriminant  analysis  by  Hao^) 
and  Bryan^,  Rao  gives  the  necessary  theoretical  discussion  and  proofs  for 
discriminant  analysis. 

The  extension  from  the  *wo -category  to  the  q-category  problem  has  been 
accomplished  by  Byran.  He  provided  a  method  of  obtaining  more  than  one  linear 
discriminant  function  and  showed  that  a  set  of  multiple  discriminant  functions 
can  be  obtained  that  exhaust  all  the  information  available  concerning  the 
separation  of  categories. 

Prior  to  the  computation  of  the  discriminant  function,  a  set  of  variables 
(words)  must  be  selected.  Since  classification  is  performed  sequentially,  one 
level  at  a  time,  from  the  highest  to  the  lowest  level  of  the  classification  tree, 
a  general  notation  can  be  expressed  in  terms  of  one  level,  as  follows: 
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Let  the  group  G  consist  of  q  categories,  denoted  by  Cjc(k  =  l,2 . q).  Each 

category  is  represented  by  pk  reference  documents,  denoted  by  Hjkfl  =  1, . . .  ,Pk)u 
All  reference  documents  in  the  group  together  contain  n  word  types,  say 

zl»  z2»  •  *  * » zn*  Let  Xyk  represent  the  frequency  of  the  i*h  word  type  in  the  jth 
document  in  the  kth  category. 

First,  the  mean  number  of  occurrences  per  document  of  the  ith  type  in  the 
kth  category  is  computed: 


(1) 


and  from  this  the  pooled  within -category  sums  of  squares  are  computed: 


q 


P 


k 


k=l j=l 


(2) 
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Next,  the  mean  number  of  occurrences  per  document  of  the  i^1  type  in  the 
entire  group  is  computed: 


1 

q 


k  =  l 


(3) 


and  from  this  the  among-category  sums  of  squares  are  computed: 


(4) 


Finally,  a  discriminant  ratio  is  computed  for  each  word  type: 


W. 

- I -  (5) 

t  V* 

k  =  l 


q 

Ai 

k  =  l 

(q-1)  Wt 


Having  defined  a  multidimensional  space  with  48  words,  the  coefficients  of 
the  discriminant  function  can  be  computed  from  a  sample  of  documents  previ  - 
ously  assigned  to  each  category.  Let  the  matrix  A  consist  of  the  among-group 
cross-products  of  deviations  of  category  means  from  the  group  mean  weighted 
by  category  sizes: 


(6) 


q 

k  =  1 

and  the  matrix  W  consist  of  the  pooled  vvithin-group  deviation  cross-products: 


w 


ih 


2  £  (Xijk  “  Xi.k)  (X; 


hjk 


-  X 


h.k, 


(7) 


k  =  1  j=l 


v.here  \  (i,h  =  1, 2, . .  ,n)  is  now  the  frequency  of  only  the  discriminating  subset 
of  n  words .  Bryan  has  shown  that  the  condition  for  maximizing  the  ratio  of  the 
among-category  sums -of-squares  to  the  pooled  within-category  sums-of-squares 
is  satisfied  by  solving  the  determinants!  equation: 


Iw"1  A  -  XI  |  =  0  (8) 

where  I  is  the  identity  matrix,  W  and  A  are  as  defined  previously,  and  X  is  any 
one  of  the  m  eigenvalues  to  be  determined.  The  eigenvector  corresponding  to  X 
prcvides  the  set  of  coefficients  for  a  disci iminant  function  which  transforms  the 
n  individual  measurements  info  a  single  value  or  discriminant  score.  The  linear 
combinations  corresponding  to  the  resulting  eigenvectors  have  the  following 
property:  the  first  linear  combination,  corresponding  to  the  largest  eigenvalue 
X  i,  maximizes  the  discriminant  criterion  in  the  sense  that  one  is  discrimina¬ 
ting  between  two  categories;  the  second  linear  combination,  corresponding  to 
the  second  largest  eigenvalue  X  2*  maximizes  the  ratio  of  the  residual  among- 
category  sums-of-squares  to  the  residual  within-category  sums-of-squares 
after  the  effect  of  the  first  has  been  removed;  and  so  forth. 

Furthermore,  the  number  of  eigenvectors  of  the  determinantal  equation  such 
that  X^  0  is  at  most  equal  to  the  smaller  of  the  two  numbers  q-1  and  n.  These 
eigenvectors  are  the  basis  for  multiple  discriminant  functions  (MDFs)  and  ex¬ 
haust  the  total  discriminative  power  of  the  variables  relevant  to  category 
separation. 
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Thu  MDKs  arc  a  powerful  tool  in  that  they  preserve  the  information  given 
by  the  variables  relevant  to  group  separation,  and  yet  allow  one  to  classify  in 
an  m -dimensional  reduced  space,  where  m  =  min(q-l,n).  Rather  than  classify 
in  the  original  11-variable  space,  the  eigenvectors  are  used  to  transform  the 
mean  vector,  Mjk  and  dispersion  matrix,  ^or  ^  category  to  the  classi¬ 
fication  or  reduced  space,  having  m  dimensions,  where  classification  can  be 
performed  more  economically.  No  loss  of  information  is  incurred  by  the  re¬ 
duction  of  48  dimensions  to  3  dimensions.  The  m  eigenvectors  of  the  transfor¬ 
mation  matrix,  V|f  are 


vtf ' 
(n,m) 


V  V 
11  21 


V,  v„ 

In  2n 


ml 


mn 


(9) 


The  transformation  equations  are: 


(m,q)  (m,n)  (n,q) 


(10) 


(m,m,k)  (m,n)  (n,n,k)  (n,m) 


(11) 


where  f,g  =  1,2,  ,..m  and  the  line  below  each  equation  indicates  the  dimensions 
of  the  matrices.  The  two  matrices  anc  Dfgk  thus  define  the  category  Ck  in 
terms  of  its  centroid  and  dispersion  in  the  discriminant  space. 

New  documents  are  classified  by  counting  all  its  words  and  retaining  only 
the  frequencies  of  the  n  discriminating  words.  These  frequencies  Xij 
(i  -- 1, 2, . . .  ,n)  for  the  j*  document  are  transformed  to  m-dimension  space  by 
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(12) 


(m,p) 


(m,n)  (n,p) 


Xf.  represents  the  f®1  coordinate  of  the  document  in  the  classification 
space.  Classification  is  accomplished  by  computing  the  distance  the  docu¬ 
ment  is  from  each  of  the  q  categories  and  computing  the  relative  probability  of 
membership  in  the  r^1  category  by 


4 =  <2fl  -  v  V1  ^  -  “fk> 

(p.q)  (m,p)  (m.q)  (m,m,k)  (m,p)  (m,g) 


(Id) 


#“,1/2  exp(-^xi) 


V  (CrlxP 


D 


2  jr 


s 


(14) 


k  = 


=1I^I 


1/8  exp(-  ~  ?j2k) 


where  pk  is  the  a  priori  probability  of  membership  in  category  k,  and  k  = 

If  2  j .  •  • , r  •  •  *  •  i  q> 

If  the  distribution  of  the  categories  coordinates  ai?e  normally  distributed 
about  the  category  centroid,  then  the  square  of  the  distance  from  the  centroid 
has  a  chi-square  distribution.  The  current  decision  rule  assigns  the 
document  to  the  category  for  which  9jr  is  the  highest. 


PROGRAM  DESCRIPTION 

A  set  of  experimental  computer  programs  has  been  written,  tested  and 
used  to  perform  document  classification  using  multiple  discriminant  functions. 

Figure  1  shows  the  overall  logic  flow  of  the  classification  programs.  The 
Frequency  Tape  Generator  selects  documents  from  a  master  document  tape  that 
are  required  for  a  particular  experiment,  and  extracts  the  frequencies  of  a 
given  set  of  words  from  each  selected  document. 
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Figure  1.  Discriminant  Analysis  Logic  Diagram 


The  primary  output  is  a  binary  document  tape  containing  the  document 
identification  and  a  list  of  up  to  50  words  and  their  frequencies .  The  program 
will  output  either  raw  or  relative  frequencies  depending  on  user  need.  A 
secondary  output  is  a  listing  of  the  words  and  their  frequencies  and  summary 
information  concerning  the  distribution  of  words  across  the  documents. 

The  discriminant  analysis  program  computes  the  means  and  dispersion  for 
each  category,  and  the  discriminant  coefficients;  it  then  transforms  the  cen¬ 
troids  of  each  category  to  the  classification  space.  The  program  will  process 
up  to  50  words,  10  categories  and  any  number  of  documents. 

The  program  reads  a  sample  document  tape  output  from  the  Frequency 
Tape  Generator  program »  It  outputs  the  discriminant  coefficients  or.  a  tape  for 
the  classification  program.  The  program  outputs  for  off-line  listing:  the  mean 
frequency  and  its  standard  deviation  of  each  word  in  each  category  and  the  total 
group;  the  total  correlation  matrix;  the  eigenvalues  and  eigenvectors  (discrimi¬ 
nant  coefficients);  and  ihe  centroids  and  dispersions  of  each  category  in  the 
original  space  and  the  reduced  space.  An  option  exists  for  a  more  detailed  out¬ 
put  of  many  intermediate  matrices . 

The  classification  program  classifies  documents  and  outputs  summary 
information  concerning  the  classification  parameters.  Each  document  is 
classified  by  transforming  the  frequencies  of  its  discriminating  words  to  the 
classification  space.  In  the  classification  space  the  distance  from  the  centroid 
of  each  category  to  the  document  is  computed.  The  document  is  classified  into 
the  nearest  category. 

The  program  also  outputs  tables  of  the  number  of  correctly  and  incorrectly 
classified  documente.  Four  tables  indicate  the  distribution  of  mis  classifications 
across  the  following  parameters:  number  of  sentences,  document  length,  num¬ 
ber  of  discriminating  types,  and  radii.  A  fifth  table  outputs  overall  summary 
information  including  Swots'  coefficients,  and  Cleverdon’s  ratios. 

Classification  of  a  set  of  sample  documents  upon  which  the  discriminant 
coefficients  are  based  or  a  set  of  independent  test  documents  can  be  accomplished. 

Complete  descriptions  of  these  programs  including  flowcharts  and  opera¬ 
ting  instructions  are  contained  in  a  separate  report,  Computer  Program 
Description  and  Instructions,  November  22,  1965. 
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Section  IV 


EXPERIMENTAL  DATA  BASE 


The  data  base  used  in  these  experiments  consisted  of  2754  abstracts  from 
the  solid  state  physics  field,  published  by  the  Cambridge  Communications  Cor¬ 
poration  (CCC),  Cambridge,  Massachusetts.  The  abstracts  had  been  classified 
by  CCC  into  subject  categories.  Correct  experimental  classification  consisted 
of  matching  the  CCC  classification. 

The  CCC  subject  structure  contained  five  categories  at  the  uppermost  level 
and  up  to  ten  levels  of  detail  in  some  categories .  Categories  selected  for  the 
experiment  contained  at  least  280  documents,  twice  the  largest  sample  size. 
This  yielded  three  categories  at  the  upper  level  and  three  categories  at  the 
second  level  (referred  to  as  the  lower  level  throughout  this  report)  as  she  .vn  in 
Figure  2.  All  documents  assigned  to  these  categories  constituted  the  document 
corpus,  consisting  of  1743  documents  at  the  upper  level  and  863  at  the  lower 
level.  Random  sampling  was  used  to  draw  documents  from  the  document  cor¬ 
pus.  Three  different  samples  of  35,  70  and  140  documents  drawn  from  each 
category  were  used  to  compute  the  statistics  required  by  the  multiple  discrimi  - 
nant  technique .  The  total  number  of  tokens  and  unique  types  in  each  category 
and  group  is  shown  in  Table  1(a)  (upper  level  category)  and  Table  1(b)  (lower 
level  category) .  The  average  document  length  in  each  category  is  also  shown . 


Figure  2.  Experimental  Solid  State  Structure 
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Table  1(a)  Upper  Level  Category  Statistics 


Category 

Sample 

Size 

Total 

Tokens 

Total 

Types 

Average  Document 
Length 

A 

35 

2,870 

871 

82.0 

M 

35 

3,389 

938 

96.8 

P 

35 

3,858 

989 

110.2 

Group 

105 

10,117 

1,850 

I 

96.4 

A 

70 

5,664 

1,281 

80.9 

M 

70 

6,696 

1,383 

95.7 

P 

70 

7,198 

1,374 

102.8 

Group 

210 

19,558 

2,549 

93.1 

A 

140 

11,583 

1,819 

82.7 

M 

140 

13,367 

1,955 

95.5 

P 

140 

14,316 

1,860 

102.3 

Group 

420 

39,266 

3,425 

93.5 
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Table  1(b)  Lower  Level  Category  Statistics 


Category 

Sample 

Size 

Total 

Tokens 

Total 

Types 

Average  Document 
Length 

P2 

35 

3,653 

917 

104.4 

P3 

35 

3,505 

873 

100.1 

P4 

35 

3,187 

844 

91.1 

Group 

105 

10,345 

1,634 

98.5 

P2 

70 

7,273 

1,332 

103.9 

P3 

70 

7,855 

1,366 

112.2 

P4 

70 

6,455 

1.202 

92.2 

Group 

210 

21,583 

2,282 

2  02.8 

P2 

139* 

14,523 

1,812 

103.7 

P3 

140 

15,360 

1,815 

109.7 

P4 

140 

13,393 

1,753 

95.7 

Group 

419 

43,276 

3,031 

103.0 

*A  processing  error  caused  the  omission  of  one  document. 


Section  V 


EXPERIMENTS 

Eight  experiments  were  performed  to  acquire  data  on  the  effects  of  the 
various  parameters.  These  experiments  fall  naturally  into  three  groups.  The 
first  group  consists  of  experiments  on  parameters  of  individual  documents  and 
includes: 

Experiment  1:  Selection  of  Discriminating  Word  Types  vs  Number  of 
Sample  Documents 

Experiment  2:  Category  Definition  vs  Number  of  Sample  Documents 

Experiment  3:  Classification  Effectiveness  vs  Document  Length 

Experiment  4:  Type- Token  Distribution  vs  Document  Length 

Experiment  5:  Number  of  Concepts  in  a  Document  vs  Number  of  Word 
Types 

The  second  group  consists  of  experiments  on  parameters  of  the  classification 
structure  and  includes: 

Experiment  6:  Classification  Effectiveness  vs  Homogeneity  of  Categories 

Experiment  7 ;  Classification  Effectiveness  vs  Level  of  the  Structure 

The  third  group  consists  of  one  experiment  testing  possible  measures  of  classi¬ 
fication  effectiveness : 

Experiment  8:  Classification  Effectiveness  Measures 

The  following  descriptions  of  each  experiment  include:  purpose,  hypotheses, 
test  statistic,  experimental  procedure,  and  results.  Discussions  of  the  meaning 
of  these  results  and  the  conclusions  affecting  further  work  are  contained  in  Sec¬ 
tions  VI  and  VII. 
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EXPERIMENT  1:  SELECTION  OF  DISCRIMINATING  WORD  TYPES  vs 
NUMBER  OF  SAMPLE  DOCUMENTS 

The  number  of  sample  documents  influences  classification  effectiveness  in 
two  ways:  first,  in  the  selection  of  the  subset  of  types  for  retention  as  the  dis¬ 
criminating  set;  and  second,  in  the  estimation  from  the  sample  documents  of  the 
population  mean  vector  and  dispersion  matrix.  This  experiment  reports  data  on 
the  first  effect.  Experiment  2  reports  on  the  second  effect. 

The  number  of  discriminating  words  selected  is  limited  computationally  by 
a  requirement  to  solve  for  the  eigenvectors  of  a  square  matrix  of  that  order. 
Thus,  the  need  exists  to  select  a  subset  of  types  such  that  the  subset  will  yield 
the  best  discrimination  between  categories,  and  such  that  an  adequate  number 
of  types  will  occur  in  each  document  to  be  classified.  A  univariate  statistic, 
the  ratio  of  the  pooled  among-cate';ory  variance  to  the  pooled  within-category 
variance,  estimates  the  discrimination  abLJ  „  of  each  type.  The  mean  frequency 
of  the  type  within  a  category  estimates  the  coverage  ability  of  the  type.  (It  J3 
uneconomical  to  retain  a  type  with  high  discrimination  ability  that  is  relatively 
rare. )  The  specific  selection  criteria  are  discussed  in  detail  in  Section  m, 
"Operational  Description.  " 

Hypothesis 

There  is  no  significant  difference  in  the  ranking  of  word  types  by  the  dis¬ 
criminant  ratio,  F  as  the  number  of  sample  documents  is  increased. 

Test  Statistic 

Spearman's  correlation  coefficient  rs  is  used  to  test  for  a  significant  differ¬ 
ence  in  the  ranking^9).  The  hypothesis  will  be  rejected  if  the  computed  value  of 
t  is  less  than  the  tabulated  value  of  t  at  a  =  .  01.  a  is  the  probability  level  be¬ 
low  which  the  hypothesis  will  be  rejected.  The  test  statistic  is 


r 

s 


1  - 


n(n2  - 1) 


t  = 


r 

s 


where  dj  is  the  difference  in  the  ranks  and  n  is  the  number  of  words. 
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Procedure 


To  observe  the  effect  of  a  change  in  the  number  of  documents,  two  sets  of 
experiments  were  performed  using  sample  sizes  of  35,  70  and  140  documents  — 
one  at  the  upper  level  consisting  of  categories  A,  M,  and  P,  and  one  at  the  lower 
level  consisting  of  P2,  P3,  and  P4.  Random  samples  of  35,  70,  and  140  docu¬ 
ments  were  drawn  from  each  category.  The  ratio  of  the  pooled  among-category 
variance  to  the  pooled  within-category  variance  was  computed  for  each  type. 

Each  type  was  sorted  and  ranked  according  to  its  ratio.  The  rank  correlation 
coefficient  was  computed  on  the  difference  in  rank  of  the  first  50  types  in  the 
35  document  sample  and  140  document  sample,  and  between  the  70  document 
sample  and  the  140  document  sample. 

Results 

Table  EL  lists  sets  of  50  discriminating  words  for  various  sample  sizes.  The 
value  of  t  for  50  degrees  of  freedom  at  a  =  0.  05  is  2.  40.  The  computed  value 
for  t  exceeded  2.  40.  Therefore,  for  ail  cases  the  hypothesis  is  rejected.  Since 
no  significant  correlation  existed  among  the  sets  of  discriminating  words  for 
various  sample  sizes,  the  set  based  on  the  140  document  sample  was  used  for 
the  other  experiments. 

Tables  m  (a),  (b),  and  (c)  contain  the  discriminant  ratio  and  mean  frequencies 
of  the  first  100  discriminating  W'ords  at  the  upper  level  for  sample  sizes  35,  70 
and  140.  The  considerable  similarity  among  the  lists  of  words  indicates  that 
this  classification  parameter  warrants  further  testing.  The  importance  of  the 
selection  of  discriminating  words  is  discussed  further  in  Section  VI,  "Homo¬ 
geneous  Categories.  " 

The  selection  criteria  also  considers  the  coverage  ability  of  each  type. 

Since  coverage  statistics  were  not  available,  the  mean  frequency  of  each  type 
was  employed  to  indicate  the  coverage.  The  validity  of  using  the  mean  as  an 
indicator  of  the  coverage  was  tested  with  a  product  moment  correlation  coeffi¬ 
cient.  A  correlation  coefficient  of  0.  9  between  the  mean  frequencies  in  the  140- 
document  sample  and  the  coverage  percentages  of  test  set  indicated  that  the  mean 
was  a  valid  indicator  of  the  coverage.  A  further  test  of  coverage  was  made  on 
the  percentage  of  documents  in  which  each  type  occurred.  A  X  2  test  revealed 
that  no  significant  difference  existed  betv’een  the  coverage  percentages  of  the 
sample  and  test  sets.  Table  IV  shows  the  coverage  percentages  for  the  test 
documents  at  both  the  upper  and  lower  levels. 
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Table  II.  Discriminating  Word  Sets  for  Various  Sample  Sizes 


t-L 

O 

35 

70 

140 

35 

70 

140 

SPIN 

MAGNET 

MAGNET 

OF 

DESCRI 

CIRCUI 

MAGNET 

RESONA 

SPIN 

EFFECT 

CIRCUI 

DESCRI 

RESONA 

SPIN 

ABSORP 

DESCRI 

TRANSI 

OPERAT 

ABSORP 

ABSORP 

RESONA 

CIRCUI 

CRYSTA 

CRYSTA 

RESIST 

PHOSP” 

FERROM 

ELECTR 

DESIGN 

FIELD 

LIGHT 

FERROM 

SATURA 

DETERM 

BAND 

OUTPUT 

ELECTR 

SEMICO 

PHOSPH 

DISLOC 

OF 

TRANSI 

CONDUC 

PARAMA 

EMISSI 

CRYSTA 

UTILIZ 

DEVICE 

ACTIVA 

LIGHT 

PARAMA 

GROWTH 

SIG  NA  L 

K 

FERRIM 

EMISSI 

EXCHAN 

OUTPUT 

VOLTAG 

DESIGN 

PHOSPH 

CARRIE 

FERRIT 

REPORT 

OUTPUT 

UTILIZ 

EMISSI 

SATURA 

SUPERC 

DESIGN 

SWITCH 

CONDUC 

PARAMA 

RESIST 

IRON 

CONDUC 

USES 

OF 

EDGE 

IRON 

INTERA 

UTILIZ 

CURREN 

MAGNET 

IRON 

EXCHAN 

RESIST 

ZONE 

EMPLOY 

PROVID 

EXCITA 

OPTICA 

LIGHT 

CHARAC 

DIFFUS 

SIGNAL 

N 

BANDS 

ANTIFE 

ACCURA 

FIELD 

GROWTH 

NEEL 

ELECTR 

SEMICO 

FOUND 

REPORT 

DISLOC 

WAVELE 

SUPERC 

LUMINE 

EXPERI 

ELECTR 

SWITCH 

SATURA 

HALL 

BAND 

TRANSI 

USED 

VOLTAG 

EQUALS 

LUME’E 

OPTICA 

FORMAT 

DEVICE 

RESONA 

LUMINE 

SUSCEP 

HALL 

CONTRO 

EFFECT 

OSCILL 

PEAKS 

MOPILE 

EXCITA 

ONLY 

SYSTEM 

INPUT 

OPTICA 

BAND 

ELECTR 

DIODES 

DISLOC 

SPIN 

FERROM 

N 

MOBILI 

SUPPLY 

THEORY 

CONTRO 

ELEMEN 

FERRIT 

WAVELE 

PROVED 

RESONA 

ELECTR 

O 

IS 

EDGE 

DENSIT 

MAGNET 

CONNEC 

FLUOR E 

SURFAC 

ANISOT 

TEMPER 

AT 

MELT 

FERRIT 

ACTIVA 

CARRIE 

RESULT 

SURFAC 

EMPLOY 

SYSTEM 

ANTIFE 

CONDUC 

B  ETWEE 

ABSORP 

MEASUR 

METHOD 

EDGE 

CURREN 

IN 

K 

EFFECT 

WITHIN 

WAVELE 

RELAXA 

AMPLIF 

DIODES 

WAS 

EVALUA 

RELAXA 

SUSCEP 

S 

ON 

RESULT 

KCL 

CENTER 

BANDS 

BASE 

GROWTH 

AMPLIF 

SENSIT 

EXCITA 

SPINS 

PROCED 

USE 

ABSORP 

HEISEN 

G 

SURFAC 

EDGES 

WERE 

PULSES 

AL 

SULFID 

RADIAT 

GAP 

PULSE 

FORMAT 

ANTIFE 

TITANA 

MICRON 

KINETI 

TEMPER 

REPORT 

DELTA 

SPINS 

IONS 

LATTIC 

COUNTE 

DIODE 

REGION 

GREEN 

DENSIT 

EXPLAI 

IN 

RECORD 

SUSCEP 

TRANSV 

INTENS 

AT 

SUPPLY 

WERE 

POSITI 

SIUCO 

ULTRAV 

DEVICE 

PARAMA 

INFORM 

GARNET 

CDS 

NICKEL 

CONSID 

AMPLIF 

PULSE 

DI 

INTERA 

F LUCRE 

WAS 

CARRIE 

DIGITA 

APPLIC 

HA  MILT 

FE 

FIELD 

RAY 

IN 

EX PONE 

MEASUR 

SPECTR 

IMPURI 

CONCEN 

SUPPLY 

WERE 

CONDUC 

WIDTHS 

IT 

BATTER 

OBSERV 

DENSIT 

WIDTH 

SULFID 

INSTHU 

RECORD 

FOUND 

ANISOT 

BARIUM 

SPLITT 

SAMPLE 

EQUA  LS 

TEMPER 

OTHER 

RECOMB 

1SOTRO 

USE 

BY 

INSTRU 
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Table  HI{a).  Mean  Frequencies  of  Discriminating  Words,  35-Document  Sample 


Word 

Cntcgory 

Category 

F 

A 

M 

— p-— 

F 

Pi 

Pi 

OF 

0.  483 

3.94 

7.  17 

7.  0 

ETCHIN 

0. 110 

0.0 

0. 14 

0.0 

effect 

o  :io<> 

0.  01. 

0.  14 

0.  77 

PHONON 

o.  no 

0.0 

0.0 

0.  14 

i)  esc  hi 

o  n.»7 

0.  69 

0  20 

0.  14 

PREVIO 

0.  110 

0.0 

0.0 

0.  14 

cmcut 

(1, 

o  77 

0.  0 

0.  0 

SWITCH 

0. 110 

0.  14 

0.0 

0.0 

ELECTS 

o.  ar.fi 

0  to 

0.  34 

1.49 

SOLID 

0. 108 

0.  03 

0. 17 

0.0 

DETER M 

o.  ar>8 

0.0 

0.46 

0.06 

DEPEND 

0.  106 

0.06 

0.23 

0.46 

disuk: 

0.  252 

0.  0 

0.  74 

0.03 

INTERA 

0.  104 

0.0 

0.03 

0.  23 

CHYSTA 

0.  2fi0 

0.  09 

1.57 

0.80 

CURREN 

0.  104 

0.  46 

0.03 

0.49 

GROWTH 

0.  249 

0.  0 

0.29 

0.0 

PER FOR 

0.  103 

0.  23 

0.0 

0.06 

OUTPUT 

0.  228 

0.37 

0.0 

0.03 

MECHAN 

0. 103 

0.  03 

0. 14 

0.34 

REPORT 

0.217 

0.0 

0.34 

0.37 

BY 

0. 101 

0.74 

1.54 

1.08 

DESIGN 

0.  20C 

0.  3  7 

0.0 

0.03 

TECHNI 

0. 100 

0.06 

0.31 

0.09 

CONDUC 

0.  196 

0.  09 

0.  02 

0.57 

OPERAT 

0. 100 

0.  26 

0.  03 

0.  09 

UTILIZ 

0.  182 

0.  20 

0.03 

0.0 

MOLECU 

0. 100 

0.0 

0.0 

0.20 

ZONE 

u.  172 

0.0 

0.31 

0.0 

THAT 

0. 100 

0.43 

0.89 

1.06 

CHARAC 

0.  164 

0.34 

0.03 

0.  11 

POSSIB 

0.099 

0.  20 

0.09 

0  0 

ACCURA 

0.  162 

0.29 

0.0 

0.0 

PROCES 

0.099 

0.  03 

0.11 

0.29 

FOUND 

0.  157 

0.0 

0.  34 

0.51 

VOLTAG 

0.098 

0.60 

0.03 

0.26 

EX  PERI 

0.  153 

0.09 

0.  11 

0.37 

WHICH 

0.098 

0.83 

0.37 

0  86 

TRANSI 

0.  151 

0.77 

0.03 

0.31 

GROWN 

0.  098 

0.0 

0.23 

0.0 

FORMAT 

0.148 

0.0 

0.26 

0.  03 

PULSE 

0.  095 

0.34 

0.0 

0.0 

CONTRO 

0.  147 

5. 37 

0.06 

0.  03 

CARRIE 

0.  035 

0.03 

0.03 

0.  26 

ONLY 

0. 145 

J.  09 

0.03 

0.29 

THERMA 

0.092 

0.0 

0.09 

0.29 

DIODES 

0.  143 

0.34 

0.0 

0.0 

CONTAI 

0.  092 

0.  06 

0.23 

0.03 

SUPPLY 

0.  143 

0-  1? 

0.0 

0.0 

SIMILA 

0.091 

0.  03 

0.09 

0.  23 

PROVID 

0.  139 

0.  20 

0.03 

0.0 

HAMILT 

0.091 

0.0 

0.0 

0.09 

DENSIT 

0.  137 

0.0 

0.31 

0.  03 

JUENTI 

0.091 

0.0 

0.0 

0.09 

TEMPER 

0. 135 

0.  14 

0.69 

0.77 

HEATIN 

0.091 

0.0 

0.09 

0.0 

RESULT 

0. 133 

0.  14 

0.54 

0.69 

WHILE 

0.091 

0.0 

0.09 

0.0 

BETWEE 

0.  133 

0.09 

0.09 

0.43 

PURIFI 

0.091 

0.0 

0.09 

0.0 

IN 

0. 128 

1.80 

2.86 

3.  17 

CAUSED 

0.091 

0.0 

0.  09 

0.0 

AMPLIF 

0.  128 

0.34 

0.0 

0.03 

PORTAB 

0.  091 

0.09 

0.0 

0.0  1 

S 

0.  125 

0.0 

0.0 

0. 11 

ADVANT 

0.  091 

0.09 

0.0 

0.0 

BASE 

0. 125 

0.  11 

0.0 

0.0 

ASSUMP 

0.091 

0.0 

0.0 

0.09 

PROCED 

0.  125 

0.0 

0.11 

0.0 

OCCUR 

0. 091 

0.0 

0.0 

0.09 

EDGES 

0. 125 

0.0 

0.  11 

0.0 

FIVE 

0.091 

0.09 

0.0 

0.0 

GAP 

0.125 

0.0 

0.0 

0. 11 

VARIAB 

0.  091 

0.  09 

0.0 

0.0 

KINETI 

0, 125 

0.0 

0.11 

0.0 

RELAXA 

0.091 

0.0 

0.0 

0.17 

LATTIC 

0.  120 

0.0 

0.  03 

0.  23 

MEMORY 

0.  091 

0.17 

0.0 

0.0 

EXPLAI 

0. 120 

0.06 

0.0 

0.  23 

MELT 

0.091 

0.0 

0.09 

0.0 

AT 

0.  120 

0.  20 

0.83 

0.97 

GROW 

0.091 

0.0 

0.09 

0.0 

DEVICE 

0.  117 

0.  20 

0.0 

0.0 

ALTERN 

0.091 

0.0 

0,0 

0.09 

CONSID 

0.  115 

9.  1! 

0.  06 

0.40 

TESTS 

0.  091 

0.09 

0.0 

0.0 

WAS 

0.  115 

0.  26 

1.00 

0.66 

PULLED 

0. 091 

0.0 

0. 09 

0.0 

FIELD 

0.  115 

'*.31 

0.0 

0.63 

BRANCH 

0.091 

0.  09 

0.0 

0.0 

IMPURl 

0.  113 

0.0 

0.34 

0.11 

RELIAB 

0.091 

0.  09 

0.0 

0.0 

IT 

o 

0.  17 

0.40 

0.66 

FURNAC 

0.091 

0.0 

0.09 

0.0 

INSTHU 

0.  Ill 

0.  17 

0.0 

0.0 

CAN 

0.  091 

0.40 

0.  26 

0.09 

SAMPLE 

0.  Ill 

0.  03 

0. 17 

0.54 

SYSTEM 

0.  090 

0.43 

0. 11 

0.  09 

USE 

0.  Ill 

0.  14 

0.  03 

0.0 

DEFOF M 

0.090 

0.0 

0.37 

0.0 
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Word 

Category 

1 

Category 

F 

A 

M 

p  |  - - 

F 

pjl 

PS 

PH 

DESCRI 

0.626 

0.84 

0.21 

0. 19 

SYMMET 

0.091 

0.01 

0.01 

0. 14 

CIRCUT 

0.406 

0.81 

0.0 

0.0 

DEPEND 

0.091 

0.07 

0.24 

0.  43 

TRANS! 

0.324 

0.84 

0.07 

0.24 

ANISOT 

0.088 

0.0 

0.  03 

0. 17 

CRYSTA 

0.235 

0.  07 

1.26 

0.91 

GERMAN 

0.087 

0.0 

0.39 

0.  09 

DESIGN 

0. 193 

*  26 

0.0 

0.01 

GROWN 

0.  085 

0.0 

0.14 

0.0 

BAND 

0. 188 

0.01 

0.0 

0.36 

FERROM 

0.085 

0.0 

0.0 

0. 14 

OF 

0. 183 

4.26 

6.  26 

6.  77 

EX  PERI 

0.  084 

0.  06 

0. 17 

0.  29 

UTIliZ 

0. 182 

0.  20 

0.0 

0.01 

INTERA 

0.084 

0.0 

0.03 

0  16 

SIGNAL 

0. 181 

0.44 

0.0 

0.01 

i  MOBILI 

0.084 

0.0 

0.01 

0  24 

VOLTAG 

0. 176 

0. 46 

0.0 

0.11 

WAVE 

0.081 

0.  06 

0.0 

0.  20 

OUTPUT 

0. 174 

0.31 

0.0 

0.0 

ACCOUN 

0.  081 

0.0 

0.03 

0. 11 

SWITCH 

0. 169 

0.36 

0.0 

0.0 

CONTRO 

0.080 

0.26 

0.04 

0.  03 

USES 

0. 164 

0. 14 

0.0 

0.0 

DIGITA 

0.080 

0. 13 

0.0 

0.01 

CURREN 

0. 161 

0.47 

0.0 

0. 17 

M 

0.  080 

0.0 

0.0 

0.21 

EMPLOY 

0. 158 

0.  21 

0.03 

0.0 

PULSES 

0.  079 

0. 17 

0.0 

0.04 

DIFFUS 

0. 156 

0.0 

0. 64 

0.  04 

OPERAT 

0.079 

0.23 

0.  03 

0.  09 

FIE  ID 

0. 153 

0. 14 

0.03 

0.64 

PROVE) 

0.079 

0.  20 

0.01 

0. 03 

RETORT 

0.141 

0.01 

0.  21 

0.30 

C 

0.  076 

0.09 

0.57 

0.30 

ELECTR 

0. 138 

0.31 

0.44 

1.07 

PURPOS 

0.076 

0.  07 

0.0 

0.0 

USED 

0. 135 

0.31 

0. 14 

0.  03 

INSTRU 

0.  076 

0. 14 

0.0 

0.0 

DEVICE 

0. 133 

0.21 

0.0 

0.0 

WIDE 

0.  076 

0.  07 

0.0 

0.0 

EFFECT 

0. 124 

0. 10 

0.17 

0.70 

SPIN 

0.075 

0.0 

0.0 

0.  23 

SYSTEM 

0. 123 

0.51 

0. 13 

0.  07 

METHOD 

0.  074 

0.  09 

0.36 

0.  17 

DISLOC 

0. 122 

0.0 

0.27 

0.03 

INFORM 

0. 073 

0. 19 

0.  03 

0.01 

THEORY 

0. 122 

0.04 

0.06 

0.31 

TAPE 

0.  073 

0.16 

0.0 

0.0 

RESONA 

0. 120 

0.04 

0.0 

0.  27 

ROOM 

0.  073 

0.0 

0.06 

0. 17 

MAGNET 

0.118 

0.66 

0.01 

0.74 

ATOMS 

0.  072 

0.0 

0.17 

0.04 

AT 

0.117 

0.223 

0.93 

1.06 

TREATM 

0.072 

0.01 

0.  21 

0.04 

SUP FAC 

0.115 

0. 07 

0.51 

0. 10 

SENSIT 

0.  071 

0.  20 

0.04 

0.01 

ABSORP 

0.113 

0.0 

0.03 

0.  36 

FOUND 

0.071 

0.  07 

0.30 

0.31 

K 

0.110 

0.0 

0.20 

0.  57 

RELATI 

0.  069 

0.  03 

0.23 

0.09 

DIODES 

0.110 

0. 16 

0.01 

0.0 

THAT 

0.  069 

0.34 

0.73 

0.  76 

ON 

0. 108 

0.33 

0.86 

0.79 

WINDIN 

0.  069 

0. 11 

0.0 

0.0 

GROWTH 

0. 105 

0.0 

0.37 

0.01 

MELTIN 

0.  069 

0.0 

0.11 

0.0 

USE 

0. 105 

0.29 

0.07 

0.09 

PHONON 

0.069 

0.0 

0.0 

0. 11 

WERE 

0. 103 

0.07 

0.51 

0. 44 

VARIOU 

0.068 

0.04 

0.03 

0. 17 

PULSE 

0. 101 

0.  21 

0.0 

0.03 

DIODE 

0.068 

0.09 

0.0 

0.0 

TEMPER 

0. 100 

0. 17 

0.61 

0.86 

DESSIP 

0.068 

0.09 

0.0 

o.b 

COUNTE 

0.099 

0.11 

0.0 

0.0 

READ 

0.  068 

0.09 

0.0 

0.0 

IN 

0. 099 

1.84 

2.71 

3. 11 

STRUCT 

0.  0CB 

0.01 

0,24 

0. 17. 

SUPPLY 

0.096 

0.13 

0.0 

0.0 

SPEED 

0.  067 

0.09 

0.0 

0.01 

PARAM 

0.  096 

0.0 

0.0 

0. 13 

ADVANT 

0.067 

0.09 

0.0 

0.  01 

AMPLIF 

0.096 

0.30 

0.01 

0.0 

SOLID 

0.  067 

0.0 

0. 17 

0.02 

CARRIE 

0.096 

0.07 

0. 10 

0.40 

STORE 

0.067 

0. 10 

0.0 

0.0 

RAY 

0.395 

0.01 

0. 13 

0.0 

VAIENC 

0.067 

0.0 

0.0 

0. 10 

CONCEN 

0.095 

0.  01 

0.46 

0  31 

AMPUT 

0.  066 

0.  11 

0.0 

0.  03 

BATTER 

0.092 

0.  09 

0.0 

0.0 

FREQUE 

0.066 

0.36 

0.0 

0.30 

RECORD 

0.091 

0.  20 

0.0 

0.01  i 

AL 

0.  066 

0.01 

0.21 

0,01 

EQUALS 

0.091 

0.01 

0. 18 

0.41 

INTERP 

0.065 

0.0 

0.01 

0. 10 

BY 

9.091 

0.63 

1.29 

0.86 

NICKEL 

0.065 

0.  01 

0.0 

0. 10 
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Table  ni(c).  Mean  Frequencies  of  Discriminating  Words,  140-Document  Sample 


Word 

Category 

Word 

Category 

F 

A 

M 

P 

F 

/u 

P3 

Av 

CIRCUI 

0.382 

0.86 

0.0 

0.01 

LOGIC 

0.  084 

0. 19 

0.0 

0.0 

DESCRI 

0.341 

0.74 

0.  27 

0.15 

ZONE 

0.084 

0.0 

0. 18 

0.02 

OPEHAT 

0.291 

0.46 

0.03 

0.03 

PLANES 

0.  081 

0.0 

0.  12 

0.01 

CRYSTA 

0.291 

0.09 

1.46 

0.69 

INTERA 

0.081 

0.0 

0.02 

0.  16 

FIELD 

0.240 

0.  12 

0.04 

0.75 

AT 

0.079 

0.35 

0.92 

1.06 

OUTPUT 

0.212 

0.45 

0.0 

0.01 

SYSTEM 

0.078 

0.41 

0.  19 

0.06 

TRANSI 

0.  192 

0.69 

0.05 

0.  27 

IMPURI 

0.  078 

0.0 

0.  28 

0.09 

DEVICE 

0. 182 

0.28 

0.01 

0.01 

GROWN 

0.077 

0.0 

0.20 

0.01 

K 

0.  177 

0.  04 

0.05 

0.  79 

REQUIR 

0.076 

0.21 

0.04 

0.  04 

DESIGN 

0.  176 

0.32 

0.01 

0.01 

EXPERI 

0.  076 

0.  11 

0.20 

0.39 

UTILIZ 

0. 173 

0.  17 

0.01 

0.0 

USE 

0.  075 

0.  25 

0.  08 

0.  05 

CONDUC 

0.  167 

0.  06 

0.04 

0.46 

INTERF 

0.075 

0.02 

0. 15 

0.0 

OF 

0.  162 

4.  16 

6.28 

6.43 

DEVELO 

0.  075 

0. 14 

0.03 

0.02 

MAGNET 

0.  162 

0.48- 

0.02 

0.84 

COMPOS 

0.  074 

0.01 

0.  21 

0.  04 

PROVID 

0. 157 

0.34 

0.03 

0.03 

DENDRI 

0.073 

0.0 

0.  16 

0.0 

SIGNAL 

0.  156 

0.  25 

0.0 

0.02 

POWER 

0.071 

0.34 

0.02 

0.  14 

GROWTH 

0.  150 

0.01 

0.49 

0.04 

DEPEND 

0.071 

0.  05 

0. 16 

0.39 

DISLOC 

0.  147 

0.0 

0.59 

0.  04 

CLOSE 

0.  071 

0.0 

0.08 

0.0 

SWITCH 

0. 136 

0.37 

0.0 

0.02 

CURREN 

0.069 

0.44 

0.  03 

0.  !4 

VOLTAG 

0.  134 

0.  55 

0.01 

0. 14 

DIODES 

0.068 

0.  13 

0.0 

0.  0 

RESONA 

0. 133 

0.  06 

0.0 

0.33 

MELTIN 

0.068 

0.0 

0.  13 

0.0 

C  .>JILL 

0.  123 

0.  26 

0.0 

0.01 

COUNTE 

0.  068 

0.25 

0.01 

0.  01 

INPUT 

0.  120 

0.  18 

0.01 

0.0 

RELIAB 

0.063 

0.09 

0.01 

0.0 

SPIN  . 

0. 114 

0.0 

0.0 

0.34 

BETWEE 

0.068 

0.  18 

0.  16 

0.41 

CONTRO 

0.  109 

0.31 

0.09 

0.01 

SPEED 

0.067 

0.  11 

0.0 

0.  03 

ELECTR 

0. 109 

0.  28 

0.46 

0.99 

COMPUT 

0. 067 

0.11 

0.0 

0.  03 

CONNEC 

0.  108 

0.  17 

0.01 

0.01 

SUITAB 

0.066 

0.  09 

0.01 

0.0 

MELT 

0.105 

0.0 

0.15 

0.0 

CALCUL 

0.065 

0.01 

0.13 

0.20 

EMPLOY 

0.104 

0.19 

0.03 

0.03 

THEORY 

0.065 

0.03 

0.09 

0.22 

MEASUR 

0.104 

C.28 

0.37 

0.79 

PER FOR 

0.065 

0.14 

0.01 

0.04 

EFFECT 

0. 103 

0.09 

0.  25 

0.53 

APPLIE 

0.063 

0.06 

0.04 

0.21 

WAS 

0. 103 

0.  19 

0,86 

0.61 

RECEIV 

0.  063 

0.  11 

0.01 

0.0 

RESULT 

0. 102 

0.  12 

0.40 

0.47 

READIN 

0.  063 

0.07 

0.0 

0.0 

AMPLIF 

0.  102 

0.30 

0.01 

0.04 

PULLED 

0.  063 

0.0 

0.  07 

0.0 

ABSORP 

0.099 

0.0 

0.  03 

0. 19 

ATTRIB 

0. 063 

0.0 

0.0 

0.  07 

PULSES 

0.098 

0. 12 

0.0 

0.0 

CAPABL 

0.063 

0.  07 

0.01 

0.0 

FORMAT 

0.095 

0.0 

0.  15 

0.02 

USED 

0,061 

0.31 

0.  11 

0.  14 

REPORT 

0.095 

0.07 

0.27 

0.33 

IMPEDA 

0.  061 

0.08 

0.0 

0.0 

DIODE 

0.094 

0.  16 

0.0 

0.0 

PORTA B 

0.  060 

0.06 

0.0 

0.0 

RECORD 

0.  992 

0.  16 

0.0 

0.01 

BECOME 

0.060 

0.0 

0.0 

0.  06 

WERE 

0.091 

0.  11 

0.60 

0.33 

GROWIN 

0.060 

0.0 

0.09 

0.  0 

INFORM 

0.090 

0.  14 

0.01 

0.0 

RELAXA 

0.  060 

0.01 

0.01 

0. 19 

PULSE 

0.088 

0.39 

0.01 

0.02 

DISCUS 

0.  060 

0.29 

0.57 

0.46 

DIGITA 

0.088 

0. 12 

0.0 

0.01 

PARAMA 

0.060 

0.01 

0,0 

0.08 

IN 

0.087 

1.79 

2.62 

3.03 

ASSUME 

0.060 

0.0 

0.  02 

0.  09 

SUPPLY 

0.087 

0.  15 

0.0 

0.0 

FIELDS 

0.059 

0.04 

0.0 

0. 16 

OBSERV 

C.  086 

0.09 

0.  29 

0.51 

CONVER 

0.059 

0  13 

0.01 

0.01 

FOUND 

0.085 

0.  C6 

0.36 

0.35 

BASE 

0.  059 

0.09 

0.01 

0.0 

TEMPER 

0.084 

0.23 

0.57 

0.87 

DIFFUS 

0.  058 

0,04 

0  39 

0.  14 

INSTRU 

0.084 

0.  14 

0.0 

0.0 

HOLE 

0. 057 

0.01 

0.0 

0.  11 

TmK1«  nr 


Pc r ccnts^c  of  Documents  oomuming  specuic  1  y pc 


Upper  Level 

Lower  Level 

Type 

%  of  Documents 

Type 

%  of  Documents 

Containing  Type 

Containing  Type 

CfRCUI 

0,099 

SUPERC 

0.  061 

DESCRl 

0.  299 

RESIST 

0.  117 

O PER AT 

0.089 

SEMICO 

0.  106 

OUTPUT 

0.  054 

HALL 

0.  057 

TRANSI 

0.  190 

E LECTR 

0.439 

DEVICE 

0.  046 

MOB!  LI 

0.064 

DESIGN 

0.059 

CARRIE 

0.  131 

PROVID 

0.  002 

CONDUC 

0.  157 

SIGNAL 

0.  050 

CURRKN 

0.  095 

SWITCH 

0.  042 

SURFAC 

0.  Ill 

VOLTAG 

0.087 

DENSIT 

0.077 

OSCILL 

0.041 

RECOMB 

0.062 

CONTRO 

0.064 

TYPE 

0.  148 

AMPUF 

0.  C38 

FIELD 

0.374 

PULSE 

0.048 

SCATTE 

0.061 

SYSTEM 

0.  125 

METHOD 

0.  146 

CRYSTA 

0.337 

MAGNET 

0.  347 

GROWTH 

0.  059 

SPIN 

0.  157 

DISLOC 

0.044 

RESONA 

0.  160 

SURFAC 

0.  130 

FERROM 

0.092 

METHOD 

0.  161 

SATURA 

0.081 

FOUND 

0.  197 

PA  RAMA 

0.084 

IMPURI. 

0.  085 

EXCIIAN 

0.  050 

DISCUS 

0.378 

FERRIT 

0.056 

DIFFUS 

0.064 

TRON 

0.  058 

STRUCT 

0.096 

INTEHA 

0.  123 

SINGLE 

0.  181 

ANTIFE 

0.  046 

CONCEN 

0.  109 

*.NISOT 

0.  083 

C 

0.  153 

RELAXA 

0.  088 

CONTAI 

0.077 

SUSCEP 

0.  056 

OXYGEN 

0.  037 

IONS 

0.085 

TECHNI 

0.091 

MOMENT 

0.  062 

FIELD 

0.  206 

ABSORP 

0.  145 

K 

0.  160 

PROS PH 

0.  053 

CONDUC 

0.  138 

EMISSI 

0.  057 

EXPERI 

0.207 

LIGHT 

0.  079 

MAGNET 

0.  226 

LUMINE 

0.  037 

RESONA 

0.088 

BAND 

0.  108 

SPIN 

0.082 

OPTICA 

0.  083 

ELECTR 

0.367 

EXCITA 

0.061 

MEASUR 

0.356 

WAVELE 

0.  046 

EFFECT 

0.  249 

EDGE 

0.  033 

RESULT 

0.  283 

BANDS 

0.  047 

REPORT 

0.213 

RADIAT 

0.050 

DEPEND 

0.  195 

MICRON 

0.  057 

OBSERV 

0.217 

INTENS 

0.084 

TEMPER 

0.353 

FLUOR E 

0.  020 

BETWEE 

0.  220 

SPECTR 

- — — - . — 

0.  146 

L_ 
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EXPERIMENT  2:  CATEGORY  DEFINITION  vs  NUMBER  OF  SAMPLE 
DOCUMENTS 

The  number  of  sample  documents  used  to  define  a  category  affects  the 
classification  performance  in  many  ways.  Some  of  these  can  be  measured  di¬ 
rectly,  while  others  can  be  measured  only  indirectly.  In  any  statistical  tech¬ 
nique  it  is  naturally  assumed  that  increasing  the  sample  size  will  yield  a  better 
estimate  of  the  parameters.  One  of  the  problems  of  multivariate  techniques  is 
that  the  sample  size  necessary  increases  with  the  number  of  variables.  Since 
the  population  statistics  are  not  known,  the  parameters  derived  from  the  largest 
sample  were  considered  to  be  estimates  of  the  true  statistics.  A  test  for  a  sig¬ 
nificant  difference  between  the  parameters  of  the  largest  sample  and  smaller 
samples  was  performed.  When  the  number  of  sample  documents  is  large  enough, 
there  should  be  no  significant  difference  in  the  parameters.  Hence  misclassi- 
fication  due  to  poor  estimates  of  the  mean  and  dispersion  should  be  reduced  to 
a  minimum. 

The  purpose  of  this  experiment  is  not  to  show  how  much  misclassification 
will  be  caused  by  an  error  in  the  estimate  of  the  parameters,  but  to  determine 
whether  an  error  exists  in  the  estimate  of  the  parameters  defining  the  category. 
When  the  number  of  sample  documents  is  increased  to  the  point  where  there  is 
no  significant  difference  in  the  estimate  of  the  parameters,  it  can  be  assumed 
that  a  larger  number  of  sample  documents  would  not  affect  the  estimated  values 
and  therefore,  that  no  further  gain  in  classification  performance  could  be 
achieved. 

Hypotheses: 

Hj:  There  is  no  significant  difference  in  the  mean  vector  of  a  category 

based  on  two  different  sample  sizes. 

H2:  There  is  no  significant  difference  in  the  dispersion  matrix  of  a  cate¬ 
gory  based  on  two  different  sample  sizes. 

Test  Statistic 

For  II|  the  test  statistic  is  Kullback’s  The  value  of  the  statistic  is 

compared  to  a  X2  table.  The  hypothesis  will  be  rejected  if  2?>  X2  at  a  =  0.  01. 

For  il2  the  test  statistic  is  also  Kullback’s  2L  The  value  of  the  statists  is 
compared  to  Fisher’s  table  of  B The  hypothesis  will  be  rejected  if  2?  is  greater 
than  the  table  value  at  a  =  0.  01. 
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Procedure 


\ 

4 

1 

1 


The  mean  vector  and  dispersion  matrix  of  each  category  were  computed 
and  transformed  to  the  reduced  space.  The  statistics  of  two  categories  at  a 
time  were  compared. 

For  the  test  of  consider  the  categories  independent  samples  with  nj 
and  n2  independent  observations  (documents)  from  k-variate  (words)  normal 
populations.  The  test  statistic  is 


/s 

21 


<vv 


(17) 


and 


d.f.  =  (r-l)K, 
where 


r  is  the  number  of  samples. 

For  the  test  H2,  consider  the  categories  as  independent  samples  with  n^ 
and  n2  observations  (documents)  from  k-variate  (words)  normal  populations. 
The  test  statistic  is 


21  =  Nx  log 


+  N2  log 


(18) 


where 


N1  N2 
S  =  N  S1  +  N  S2 


N1  =  ni  -  ! 


Nrt  =  n0  -  1 
2  2 


and 


N  =  N1  +  N2* 
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Enter  the  /32  table  with  values  of  02  and  the  degrees  of  freedom  computed  by 
the  following  expressions: 


(2k3  +  3k2  -  k) 
12 


d.f. 


k  (k+ 1) 
2 


(19) 


Results 

The  experiment  was  conducted  on  all  the  categories  at  both  the  upper  and 
lower  levels.  The  sample  sizes  were  70  and  140  documents  in  all  cases. 

The  value  of  X2  at  a  =  0. 01  is  9. 2.  Table  V  shows  that  a  stable  estimate 
of  the  mean  vectors  was  obtained  for  some  of  the  categories.  The  experiment 
tested  for  a  significant  difference  between  samples  of  70  and  140,  which  means 
when  the  null  hypothesis  is  accepted  that  a  sample  of  70  is  sufficient.  The  test 
offered  no  information  concerning  the  140-document  sample  itself.  Information 
concerning  the  140-document  sample  could  be  obtained  by  testing  it  against  a 
larger  sample  or  all  the  available  documents. 


Table  V.  Test  of  Category  Means 


Upper  Level 

Lower  Level 

A 

M 

P 

P2 

P3 

P4 

21 

decision 

1. 11 
accept 

24.71 

reject 

1.  67 
accept 

1.68 

accept 

39.  68 
reject 

10.53 

reject 

The  value  of  02  for  acceptance  of  the  hypothesis  at  a  =  0. 01  is  11. 3.  Table 
VI  shows  that  considerable  difference  existed  between  the  dispersions.  The 
dispersion  matrices  in  the  original  variable  space  contain  48  x  48  elements,  and 
thus  may  require  a  larger  number  of  documents  to  obtain  a  stable  estimate. 

It  is  interesting  to  note  that,  as  the  number  of  sample  documents  increased,  the 
dispersion  increased.  While  this  might  imply  that  classification  performance 
would  decrease  as  the  number  of  sample  documents  increased,  additional  in¬ 
formation  on  this  parameter  reported  in  Section  VI  indicates  that  improvement 
in  classification  performance  can  be  expected  as  sample  size  is  increased. 
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Table  VL  Test  of  Category  Dispersions 


Upper  Level 

Lower  Level 

A 

M 

P 

p2 

P3 

P4 

2t 

decision 

286.4 

reject 

295.3 

reject 

271.5 

reject 

316.5 

reject 

310.2 

reject 

345.  8 
reject 

EXPERIMENT  3:  CLASSIFICATION  EFFECTIVENESS  vs  DOCUMENT  LENGTH 

The  purpose  of  this  experiment  was  to  assess  the  effect  of  variation  of  docu¬ 
ment  length  on  classification  performance  and  to  give  some  indication  of  the 
minimum  length  that  the  technique  can  handle.  Since  the  data  base  was  con¬ 
fined  to  abstracts  with  a  range  of  20  to  250  tokens,  considerable  data  was  col¬ 
lected  concerning  very  short  documents  and  the  variation  in  this  range. 

Hypotheses 

Hj:  There  is  no  significant  difference  in  the  mean  number  of  tokens  in  each 
document  between  the  correctly  classified  documents  and  the  incorrectly 
classified  documents  in  each  category. 

H2:  There  is  no  significant  difference  in  the  mean  number  of  sentences  in 
each  document  between  the  correctly  classified  documents  and  the  incorrectly 
classified  documents  in  each  category. 

Test  Statistic 

Student's  t  was  used  to  test  the  hypotheses  for  equality  of  the  two  sample 
means,  with  equal  variances,  and  unequal  sample  sizes.  The  test  statistic  is: 


t 


(20) 


(21) 


(22) 


2  -  h.  ~ 11 S1 1  ^  ~  a± 

3  (nj  -  1)  *  (n2  -  1) 

The  hypotheses  will  be  rejected  if  the  computed  value  of  t  is  greater  than  the 
tabulated  value  of  t  at  a  =0.01. 

Procedure 

The  mean  number  of  tokens  and  the  standard  error  of  the  mean  of  the  cor¬ 
rectly  and  incorrectly  classified  documents  in  each  category  was  computed. 

Let  and  x2.  Sj2  and  s2“,  and  nj  and  n2  denote  the  sample  means,  variances, 
and  sizes,  respectively.  The  sets  of  correctly  and  incorrectly  classified  docu¬ 
ments  were  considered  to  be  independent  samples  from  normal  populations 
having  a  common  but  unknown  variance.  An  F  test  was  used  to  establish  that 
the  variances  were  equal. 

The  degrees  of  freedom  (n  -  1)  for  each  category  varied  from  124  to  864. 

Since  the  change  in  t  in  this  range  was  very  slight,  the  value  of  t  for  100  degrees 
of  freedom  was  used  for  all  categories.  The  value  of  t  for  100  degrees  of  free¬ 
dom  at  a  =0.  01  is  2.  58. 

The  experiment  was  conducted  on  the  sample  and  test  documents  of  all  the 
categories  at  both  the  upper  and  lower  levels.  The  number  of  sample  docu¬ 
ments  used  to  compute  the  discriminant  coefficients  was  140. 

Results 

A  summary  of  the  results  concerning  is  shown  in  Table  VII.  The  results 
indicated  that  there  was  no  significant  difference  in  the  means.  Differences  as 
large  as  those  obtained  can  be  attributed  to  sampling  variation  alone.  Therefore, 
in  this  set  of  documents,  the  length  of  the  document  does  not  have  a  very  signifi¬ 
cant  effect  on  performance.  The  influence  of  document  length  on  other  param¬ 
eters  that  do  significantly  effect  performance  is  discussed  in  Section  VI. 

A  summary  of  the  results  concerning  H2  is  shown  in  Table  Vm.  The  value 
of  t  for  this  test  is  also  2.  58.  The  results  indicated  that  there  was  no  signifi¬ 
cant  difference  in  the  means.  Therefore,  in  this  set  of  documents,  the  number 
of  sentences  appeared  to  have  no  significant  effect  on  performance.  The  insig¬ 
nificant  difference  between  the  means  of  the  sample  set  and  the  test  set  indicates 
that  the  samples  were  very  representative  of  their  populations. 
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Table  VII.  Test  of  Document  Length 


Upper  Level 

Lower  Level 

A 

M 

P 

P2 

P3 

P4 

Sample 

Correct, 

83.2 

96.0 

105.8 

103.6 

111.0 

97. 1 

Znoorrect,  x2 

79.9 

93.2 

82.0 

101.4 

89.3 

88.5 

t 

0.  40 

0.34 

2.  2 

0. 14 

1.3 

0. 95 

Decision 

accept 

accept 

accept 

accept 

accept 

accept 

Test 

Correct,  Xj 

79.5 

95.6 

108.4 

101.6 

108. 1 

95.6 

Incorrect,  x2 

85.3 

80.3 

92.  7 

108.7 

82.  1 

88.3 

t 

0. 93 

2.4 

4.  5 

0.  72 

1.8 

0.  76 

Decision 

accept 

accept 

reject 

accept 

-  -  ■  ■ 

accept 

accept 

Table  VE  Test  of  Number  of  Sentences 


Upper  Level 

Lower  Level 

A 

M 

P 

P2 

p3 

P4 

Sample 

Correct,  Xj 

Incorrect, 

t 

Decision 

4.5 

4.3 

0.48 

accept 

5.3 

4.9 

0.89 

accept 

5.5 

4.3 

2.3 

accept 

5. 1 

4.7 

0.66 

accept 

5.5 

4.9 

0.  68 

accept 

5.2 

4.5 

1.4 

accept 

Test 

Correct, 

Incorrect,  x2 

t 

Decision 

4.5 

4.  8 

0.89 

accept 

5. 1 

4.3 

2.3 

accept 

5.5 

4.9 

3,3 

accept 

5.3 

5.4 

0.30 

accept 

5.5 

4.3 

2. 1 

accept 

5.1 

4.5 

1. 1 

accept 
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EXPERIMENT  4:  TYPE-TOKEN  DISTRIBUTION  vs  DOCUMENT  LENGTH 


One  problem  encountered  in  many  textual  data  handling  applications  is  de¬ 
termining  how  many  documents  are  needed  to  get  the  system  going.  The  prob¬ 
lem  has  two  aspects:  cost  and  statistical  reliability.  Expressions  exist  for 
computing  the  sample  size  required  for  a  given  confidence  level  when  the  dis¬ 
tribution  of  a  random  variable  is  known.  However,  when  using  this  technique 
the  distribution  of  words  is  not  known  ar.d  cither  techniques  must  be  employed. 

In  addition  to  the  lack  of  a  distribution  function,  textual  applications  fre¬ 
quently  have  the  missing  variable  problem:  the  same  words  do  not  occur  in 
every  document.  This  experiment  was  concerned  with  the  interrelationship  of 
the  number  of  documents  and  their  lengths  and  the  occurrence  of  word  types. 

One  of  the  steps  of  the  classification  technique  is  the  selection  of  a  subset 
of  words  on  which  classification  is  based.  The  selection  criteria  is  based  on 
a  univariate  statistic  computed  from  a  sample  of  documents.  The  sample  must 
be  large  enough  to  ensure  not  only  that  the  statistic  is  reliable  but  also  that  ail 
the  significant  words  have  occurred.  If  the  sample  is  too  small,  words  that  are 
truly  good  discriminators  may  not  have  appeared  yet. 

The  appearance  of  new  word  types  may  be  analyzed  by  their  arrival  rate. 
The  arrival  rate  may  be  measured  either  on  a  per  document  basis  or  on  a  per 
token  basis.  For  the  statistical  technique  to  be  feasible,  the  arrival  rate  must 
decrease  as  the  number  of  documents  or  the  number  of  tokens  increases.  In 
fact,  for  the  technique  to  be  economical  the  arrival  rate  must  approach  a  small 
number  fairly  rapidly.  The  arrival  rate  will  never  approach  zero,  since  many 
documents  will  contain  a  type  that  is  used  only  in  that  document.  This  is  not  a 
problem,  however,  as  long  as  a  subset  of  words  can  be  found,  each  of  which 
occurs  in  some  portion  of  the  documents ,  and  that  the  subset  as  a  whole  offers 
sufficient  coverage  of  each  document. 

If  one  has  long  documents  it  would  appear  that  not  as  many  documents  are 
required  to  obtain  an  adequate  sample  size  as  with  short  documents.  But,  the 
relative  terms  'long"  and  "short"  need  some  quantitative  bounds.  The  range 
of  documents  in  the  solid  state  data  base  is  from  20  to  250  tokens.  The  purpose 
of  this  experiment  was  to  determine  if  any  significant  difference  exists  in  the 
number  of  types  obtained  from  samples  of  long  and  short  documents  exists. 

The  documents  were  grouped  into  two  sets:  those  having  100  or  more  tokens 
and  those  having  less  than  100  tokens. 
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■  This  experiment  provided  information  on  the  interrelation  of  the  length  of 

documents  and  the  number  of  documents,  the  cumulative  distribution  of  unique 

;  types,  and  the  arrival,  rate  of  new  types. 

\ 

Hypothesis 

There  is  no  significant  difference  in  the  relationship  of  the  cumulative  num¬ 
ber  of  types  to  the  cumulative  number  of  tokens  when  the  individual  document 
lengths  arc  changed. 

Test  Statistic 

o 

X  is  used  to  test  for  a  significant  difference  between  the  cumulative  types 
for  short  documents  and  the  cumulative  types  for  long  documents. 

Procedure 

Figure  3  shows  the  subset  of  the  structure  selected  to  perform  this  experi¬ 
ment.  Categories  containing  a  sufficiently  large  number  of  documents  were 
chosen.  To  minimize  variation  caused  by  different  subject  matter,  subcate¬ 
gories  of  major  categories  were  selected.  All  the  documents  of  each  category 
were  then  partitioned  into  two  sets.  Set  A  contained  documents  of  100  word 
tokens  or  more.  Set  B  contained  documents  of  less  than  100  word  tokens. 

The  words  in  each  document  were  counted  and  the  cumulative  number  of 
tokens  and  types  was  computed.  Linear  interpolation  was  performed  on  the 
data  where  necessary  in  order  to  make  the  comparisons.  X2  was  computed  on 
the  differences  between  values  of  the  short  and  long  documents.  The  degree  of 
freedom  is  one  less  than  the  number  of  documents  being  compared. 

Results 

The  data  in  Table  IX  indicates  that  no  significant  difference  existed  between 
long  and  short  documents  at  the  third  and  fourth  levels  and  that  a  significant 
difference  existed  between  long  and  short  documents  at  the  first  and  second 
levels.  The  difference  existing  at  the  upper  levels  may  be  due  to  variation  of 
two  parameters  not  controlled  by  this  experiment.  At  the  first  level  documents 
were  drawn  from  five  different  categories,  while  at  the  P  levels  documents  were 
drawn  only  from  the  P  category.  It  could  be  assumed  that  a  relation  exists  be¬ 
tween  the  number  of  different  categories  and  the  number  of  different  words. 

The  other  parameter  is  the  variation  in  average  document  length.  The  average 
length  of  documents  in  A,  M,  P  is  83,  96,  and  102  tokens,  respectively.  This 
deviation  would  cause  more  A  documents  to  be  placed  in  the  below- 100  set  and 
more  P  documents  to  be  placed  in  the  above- 100  set. 
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Experimental  Structure  for  Type- Token  Test 


Table  EX.  Cumulative  Word  Typos 


Tokerui 

P301A 

P301B 

P302A 

P302B 

500 

225 

232 

228 

239 

600 

257 

267 

265 

271 

700 

294 

301 

299 

299 

800 

326 

336 

332 

322 

900 

355 

363 

355 

342 

1,000 

385 

395 

378 

373 

1,500 

509 

531 

494 

489 

623 

734 

626 

754 

591 

679 

585 

672 

3,000 

810 

825 

742 

3,500 

846 

891 

803 

4,000 

910 

960 

887 

4, 500 

971 

1004 

949 

S,000 

1041 

1003 

5,500 

1102 

1069 

6,000 

1164 

1123 

6,500 

1206 

1197 

7, 000 

1255 

1296 

7, 500 

1299 

1338 

8,000 

1349 

1377 

8.500 

1390 

9, 000 

1438 

9,500 

1475 

10, 000 

1504 

iW|  vvv 
20, 000 
25, 000 
30,000 
35, 000 
40, 000 
45,000 
50, 000 
55, 000 
60, 000 
65, 000 
70, 000 
75, 000 
80, 000 
85, 000 
90,000 
95, 000 
100, 000 


Categoi 

2 _ _ _ 

P2A 

P2B 

P3A 

P3B 

PA 

PB 

SSA 

SSB 

214 

234 

224 

251 

232 

236 

243 

257 

258 

273 

274 

285 

263 

274 

281 

293 

287 

304 

298 

312 

294 

303 

311 

326 

323 

333 

327 

346 

320 

332 

340 

354 

357 

350 

353 

375 

347 

359 

372 

384 

390 

380 

379 

401 

377 

385 

398 

419 

519 

507 

500 

502 

517 

519 

525 

572 

624 

622 

607 

632 

654 

639 

637 

694 

714 

710 

691 

714 

737 

729 

745 

789 

798 

782 

808 

794 

821 

812 

852 

862 

681 

863 

866 

866 

902 

898 

947 

962 

931 

918 

918 

925 

954 

994 

1009 

1039 

1015 

978 

969 

983 

1012 

1053 

1065 

1112 

1072 

1050 

1029 

1043 

1067 

1134 

1167 

1173 

1110 

1110 

1082 

1083 

1119 

1174 

1219 

1248 

1151 

1160 

1150 

1135 

1169 

1238 

1274 

1339 

1198 

1217 

1187 

1193 

1208 

1300 

1331 

1425 

1259 

1272 

1240 

1227 

1257 

1353 

1369 

1492 

1301 

1326 

1287 

1275 

1343 

1396 

1412 

1565 

1343 

1351 

1335 

1332 

1382 

1454 

1448 

1628 

1372 

1402 

1373 

1429 

1496 

1493 

1671 

1403 

1439 

1403 

1460 

1546 

1540 

1714 

1450 

1466 

1435 

1490 

1579 

.572 

1755 

1495 

1500 

1482 

1523 

1618 

1615 

1800 

1836 

1810 

1861 

1927 

1969 

2173 

2102 

2119 

2159 

2305 

2507 

2343 

2386 

2541 

2791 

2525 

2599 

2828 

3018 

2677 

3010 

3217 

2851 

3192 

3377 

2997 

3363 

3554 

3142 

3495 

3699 

3262 

3626 

3829 

3356 

3752 

3967 

3498 

3890 

“.084 

4057 

4214 

4146 

4359 

4253 

4469 

4336 

4603 

4445 

4724 

4541 

4850 

_ 

4638 

Computed  X  2 
Table  X  2 
Decision 


9.75 

21.03 

Accept 


1.91 
5.51 
Acorot 


10.  58 
36. 17 
Accept 


12.  262 

30.14 

Accept 


56.05 
40. 11 
Reject 


402. 18 
55.80 
Reject 


Additional  data  was  obtained  which  substantiates  the  conclusion  of  no  sig¬ 
nificant  difference  for  homogeneous  categories.  The  cumulative  distributions 
of  P3A  and  PA  were  compared  with  the  distribution  from  a  random  sample  of 
140  documents  in  each  category.  The  random  sample  was  drawn  independent 
of  document  length.  The  chi-square  values  were  less  than  the  criterion,  and 
the  hypothesis  of  no  significance  was  therefore  accepted. 

The  experiment  also  revealed  that  the  shape  of  the  cumulative  distribution 
curve  is  fairly  independent  of  the  subject  category.  The  only  variation  was 
regular,  viz. ,  in  moving  up  the  structure  toward  n  more  general  category  the 
number  of  types  increased  slightly.  The  cumulative  number  of  types  is  there¬ 
fore  a  function  of  the  total  number  of  tokens  observed  rather  than  the  number 
of  documents. 

From  the  SSB  column  of  Table  DC  it  can  be  observed  that  the  arrival  rate 
of  new  types  decreased  considerably.  In  the  first  10,000  tokens  1800  types 
had  arrived,  whereas  in  ne  80,000  to  90,000  interval  only  255  types  had  ar¬ 
rived.  Since  the  average  SSB  document  length  is  72,  and  each  document  con¬ 
tains  1.83  new  types,  96  percent  of  the  types  in  each  document  has  already 
been  observed.  The  arrival  rate  is  approaching  a  small  number,  thus  con¬ 
firming  that  the  words  selected  from  a  sample  will  be  representative  and  will 
not  adversely  affect  classification  performance. 


EXPERIMENTS:  NUMBER  OF  CONCEPTS  IN  A  DOCUMENT  vs  NUMBER 
OF  WORD  TYPES 

The  number  of  concepts  in  a  doom  ent  is  important  in  a  classification  sys¬ 
tem,  since  (documents  discussing  more  than  one  subject  can  be  placed  only  in 
the  category  dealing  with  the  major  or  primary  subject,  or  in  several  categories, 
one  for  each  subject.  If  documents  dealing  with  more  than  one  subject  have  a 
greater  number  of  word  types  for  a  given  document  length,  then  the  probability 
of  classifying  them  correctly  into  several  categories  based  on  a  fixed  set  of 
discriminating  words  will  be  greater,  since  more  of  the  discriminating  words 
can  be  expected  to  appear.  On  the  other  hand,  if  the  number  of  word  types  for 
a  given  document  length  is  the  same  for  single-and  multiple-concept  documents, 
then  longer  multiple-concept  documents  will  be  needed  to  achieve  the  same 
classification  accuracy  as  with  single-concept  documents. 

This  experiment,  therefore,  was  designed  to  investigate  the  relationship  of 
the  number  of  word  types  in  the  document  to  the  number  of  concepts  in  the  docu¬ 
ment.  The  number  of  concepts  in  the  document  was  taken  as  the  number  of  cate¬ 
gories  assigned  to  it  by  Cambridge  Communications  Corporation. 
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Hypothesis 


For  a  given  number  of  tokens  ,  there  is  no  significant  difference  in  the  num¬ 
ber  of  types  between  singly-classified  documents  and  multiply-classified  documents. 

Test  Statistic 

Since  the  type-token  distribution  is  not  known  *  the  non-parametric  U  Test 
was  usedO-1).  This  test  is  based  on  the  sums  of  ranks  and  is  useful  in  testing 
for  the  equality  of  two  populations.  Let  Xp  X2. .  •  •  ,  Xm  be  a  sample  from  a 
population  with  frequency  function  f(X),  and  let  Yp  Y2,  .  • .  ,  Yn  be  a  sample 
from  a  population  with  frequency  function  f(Y).  Then,  to  test  the  hypothesis 
that  f(X)  =  f(Y),  rank  the  combined  set  of  values  Xj  (i=  1,  2,  . . .  ,  m)  and 
Yj  (j  =  1,  2,  ...  ,  n),  in  order  of  increasing  magnitude.  Let  T  denote  the  sum 
of  the  ranks  of  the  Y  values.  Then  the  statistic 


U  -  mn  +  -  T 


is  approximately  normally  distributed  when  rr<  and  n  are  both  greater  than  8. 
Furthermore, 


£  [U]  = 


mn  (m  +  n  +  1) 
12 


(25) 


Procedure 

Thirteen  cases  were  tested  to  establish  the  hypothesis.  In  each  case,  a 
fixeu  document  length  was  chosen,  and  documents  of  that  length  were  examined 
to  see  whether  they  were  singly,  doubly  or  triply  classified.  Comparisons  were 
planned  to  be  made  when  there  were  eight  or  more  documents  in  ail  sets.  How¬ 
ever,  because  of  the  diificulty  of  finding  enough  documents  of  the  same  length  in 
the  relatively  small  data  base  cuing  used,  fewer  than  eight  documents  were 
accepted  in  some  cases. 

After  the  documents  had  been  selected  for  each  case,  they  were  ranked, 
and  the  values  of  U.  E  [u]  and  Ojf  were  computed.  The  hypothesis  was  ac¬ 
cepted  at  the  1  percent  level  as  long  as: 
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Results 

The  data  used  and  results  obtained  are  displayed  in  Table  X.  In  all  cases 
tested,  the  hypothesis  was  verified:  for  documents  of  the  same  length,  there 
was  no  difference  in  the  number  of  types  between  documents  with  different  num¬ 
bers  of  concepts. 

The  mean  document  length  and  mean  number  of  types  for  singly,  doubly, 
and  triply  classified  documents  were  computed  and  are  given  below: 

Class  Mean  Length  Mean  No.  of  Types  Types/Length 

Singly  classified  93.  13  61.07  0.66 

Doubly  classified  96. 35  62.  18  0. 65 

Triply  classified  110.43  69.15  0.63 

Thus,  although  the  density  of  types  is  not  significantly  different  among  the  three 
classes  of  documents,  the  lengths  do  increase  with  the  number  of  concepts,  and 
the  number  of  types  increases  as  would  be  expected  in  longer  documents.  Also, 
as  expected,  the  ratio  of  the  two  drops  as  the  length  increases.  However,  the 
wide  range  in  document  lengths  used  produces  a  large  variance,  so  the  increase 
in  mean  lengths  is  not  significant. 


EXPERIMENT  6:  CLASSIFICATION  EFFECTIVENESS  vs  HOMOGENEITY  OF 
CATEGORIES 

Lack  of  homogeneity  of  a  category  represented  by  the  statistical  model  may 
be  caused  by  several  diverse  factors.  The  factors  involve  the  definition  of  the 
subject  category,  the  documents  assigned  to  it,  the  words  used  by  the  author, 
the  words  selected  as  the  discriminating  subset,  and  the  estimates  of  the  sta¬ 
tistical  parameters  defining  that  category. 

The  conceptual  definition  of  a  category,  such  as  Solid  State  Devices,  may 
include  concepts  also  included  in  the  definition  of  another  category.  The  docu¬ 
ments  assigned  to  a  category  may  also  be  assigned  to  another  category  and  thus 
contain  words  representative  of  more  than  one  category.  Therefore  the  mean 
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Number  of  Word  Types  vs  Number  of  Concepts 
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frequencies  among  categories  may  not  differ  significantly  in  some  cases.  The 
effect  of  these  factors  in  the  model  should  be  reflected  in  the  estimates  of  the 
mean,  dispersion,  and  position  of  the  ellipses  representing  the  categories. 

The  purpose  of  this  experiment  was  to  determine  if  any  decrease  in  classi¬ 
fication  performance  could  be  attributed  to  lack  of  homogeneity  as  measured  by 
the  magnitude  of  the  dispersion  matrix. 

Hypothesis 

There  is  no  significant  difference  in  the  ranking  of  the  classification  effec¬ 
tiveness  and  the  homogeneity  of  a  category. 

Test  Statistic 

Spearman’s  correlation  coefficient  rs  is  used  to  test  for  a  significant  dif¬ 
ference  in  ranking. 

The  test  statistic  is  given  by  equations  (15)  and  (16).  The  hypothesis  will 
be  rejected  if  the  computed  value  of  t  is  less  than  the  tabulated  value  of  t  at 

Qr  ~  0.  01. 


Procedure 

The  determinant  of  the  dispersion  matrix  of  each  category  in  the  discriminant 
space  was  computed.  The  categories  were  ranked  in  descending  order  of  their 
determinants.  The  percentage  of  correct  classifications  among  the  test  docu¬ 
ments  for  each  category  was  computed,  and  the  categories  were  ranked  in  order 
of  increasing  accuracy.  For  this  test,  categories  from  the  upper  and  lower 
levels  were  considered  together.  The  classification  results  were  based  on  dis¬ 
criminant  coefficients  computed  from  the  140-document  sample. 

Results 


Table  XL  Test  of  Homogeneity 


Rankjj 

Category 

Determinant 

Effectiveness 

Ranke 

1 

A 

0.  1011 

0.88 

5 

2 

P 

0. 0994 

0.  73 

1 

3 

M 

0.0494 

0.  73 

2 

4 

P3 

0.  0333 

0.92 

6 

5 

P2 

0.  0332 

0.87 

4 

6 

P4 

0.  C299 

0.  35 

3 
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The  value  of  t  with  4  degrees  of  freedom  at  or  =  0. 01  is  4.  G.  The  hypothesis  [ 

was  rejected  since  the  computed  value  of  t  is  0. 17.  Thus,  the  relative  size  of  j 

the  dispersion  matrix  in  itself  appeared  to  have  no  significant  affect  on  performance. 

The  determinants  of  all  the  lower  level  categories  were  considerably  smaller  | 

than  those  of  the  upper  level.  Two  factors  that  are  independent  of  level  con¬ 
tributing  to  the  dispersion  matrix  were  higher  frequencies  and  words  occurring 
in  more  than  one  category.  An  important  factor  related  to  homogeneity  causing 
a  decrease  in  performance  appeared  to  be  the  position  of  each  word  in  the  dis¬ 
criminant  space.  Some  of  the  words  were  extremely  far  from  the  centroids  of 
their  categories  and  in  fact  very  close  to  the  centroid  of  another.  Influence  of 
the  position  of  words  on  performance  is  discussed  in  Section  VI,  "Homogeneity 
of  Categories.  " 

EXPERIMENT  7:  CLASSIFICATION  EFFECTIVENESS  vs  LEVEL  OF  THE 
STRUCTURE 

This  experiment  was  intended  to  test  the  ability  of  the  technique  to  classify 
at  more  than  one  level.  Since  the  model  is  stated  in  general  terms ,  it  should  be 
insensitive  to  a  change  in  level.  The  technique  should  be  useful  at  several  levels 
of  detail,  requiring  only  that  a  sufficient  number  of  sample  documents  exist  to 
estimate  the  statistical  parameters. 

Hypothesis 

There  is  no  significant  difference  in  the  probable  error  matrix  from  level 
to  level. 

Test  Statistic 

o 

X  is  used  to  test  for  a  significant  difference  between  3x3  tables  of  ob¬ 
served  frequencies  and  expected  frequencies.  The  number  of  degrees  of  free¬ 
dom  is  6. 

Procedure 

The  probable  error  matrix  was  computed  for  all  categories  at  both  levels. 

The  sample  size  was  140  documents  for  each  category.  The  X2  test  requires 
integers,  therefore  the  percentages  were  multiplied  by  100. 

To  compare  observed  frequencies  with  expected  frequencies,  some  similarity 
must  exist  between  the  populations.  No  such  similarity  existed  between  subject 
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headings  at  one  level  and  those  at  another  level.  Therefore,  the  statistical  prop¬ 
erties  of  the  categories  were  tested.  Since  the  sequence  of  categories  is  arbi¬ 
trary  (P2,  P3,  P4;  P4,  P3,  P2),  the  lower  level  categories  were  assigned  the 
following  sequence  for  this  test:  P3,  P2,  P4.  The  criteria  of  ordering  was 
based  on  the  location  of  the  categories  in  the  discriminant  space.  The  first 
category  is  the  one  which  is  on  an  axis  by  itself;  the  second  category  is  that  one 
of  the  remaining  two  nearest  the  origin. 

Results 


Table  XII.  Test  of  Lower  Level 


Upper  Level 

Lower  Level 

A 

M 

£ 

P3 

P2 

P4 

A 

1.  00 

0.  00 

0.00 

P3 

1.0 

0.  00 

0.00 

M 

0.  01 

0.93 

0.  06 

P2 

0.  0 

0.91 

0.  09 

P 

0.  03 

0.03 

0.  93 

P4 

0.0 

0.  01 

0.99 

The  value  of  X2  at  a  =  0. 05  for  six  degrees  of  freedom  is  11. 07.  The 
hypothesis  is  accepted  siuce  the  computed  X2  of  7.260  is  less  than  11.07. 


This  test  did  not  provide  adequate  information  about  the  effects  of  level  of 
the  structure  on  effectiveness.  Therefore,  in  addition  to  this  test,  the  lower 
level  categories  were  included  in  many  other  experiments.  Corresponding 
results  between  levels  are  shown  in  the  tables  in  the  Appendix.  In  general, 
better  results  were  achieved  at  the  lower  level  than  at  the  higher  level  be¬ 
cause  of  the  set  of  discriminating  words  used.  The  position  of  the  discriminating 
words  in  the  discriminant  space  appears  to  have  a  more  significant  effect  on 
classification  performance  than  the  level. 


EXPERIMENTS:  CLASSIFICATION  EFFECTIVENESS  MEASURES 

Measures  of  classification  effectiveness  are  needed  in  the  following  phases: 
in  the  testing  phase  not  only  to  measure  overall  classification  effectiveness  but 
also  to  measure  changes  in  effectiveness  due  to  a  change  in  a  parameter;  in  the 
implementation  phase  to  test  whether  a  new  set  of  data  meets  the  assumptions 
required  for  the  technique;  and  in  the  operational  phase  to  continually  monitor 
the  system  performance. 


This  experiment  tested  two  measures  of  classification  effectiveness.  One 
is  the  probable  error  matrix  which  requires  the  assumption  that  the  data  is 
multivariate  normal,  and  the  other  is  the  actual  error  matrix  of  the  sample 
documents  which  does  not  require  this  assumption.  The  value  of  the  selected 
measure  would  be  to  indicate  the  expected  error  in  classifying  new  documents 
and  to  aid  in  making  implementation  decisions  concerning  a  specific  set  of  data. 

Hypotheses 

H^:  There  is  no  significant  difference  between  the  probable  error  matrix 

and  the  test  document  error  matrix. 

H2:  There  is  no  significant  difference  between  the  sample  document  error 

matrix  and  the  test  document  error  matrix. 

Test  Statistic 

O 

X  is  used  to  test  for  a  significant  difference  between  3x3  tables  of  ob¬ 
served  frequencies  and  expected  frequencies.  The  number  of  degrees  of  free¬ 
dom  is  six. 

Procedure 

The  probable  error  matrix  was  computed  by  considering  the  centroid  repre¬ 
senting  a  category  as  a  random  point  in  the  discriminant  space.  The  probability 
of  membership  in  each  category  was  computed  by  equation  (14).  In  the  ideal 
case,  the  probability  of  its  being  in  its  own  category  would  be  1. 0.  Thus  all  the 
diagonal  elements  of  the  matrix  would  be  1. 0  and  all  the  off-diagonal  elements 
would  be  0. 0.  In  the  real  case,  the  off-diagonal  elements  would  indicate  the 
amount  of  overlap  of  categories  and  hence  the  probable  error  of  raisciassifying 
into  that  category. 

The  sample  and  test  error  matrices  were  derived  from  the  actual  number 
of  documents  classified  by  the  technique.  Since  the  number  of  documents  in 
each  category  in  the  sample  set  and  test  set  differ,  they  have  been  normalized 
by  the  number  of  the  documents  in  the  category.  The  X2  test  requires  integers, 
therefore  the  percentages  were  multiplied  by  100. 
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Results 


Table  XIII.  Error  Matrices 


Upper  Level 

Probable  Error 

_ 

Sample  Error 

Test  Error 

A 

M 

P 

A 

M 

P 

A 

M 

P 

A 

1. 00 

0.00 

0.00 

0.88 

0.03 

0.09 

0.88 

0.05 

0.07 

M 

0.01 

0.93 

0.06 

0.04 

0.82 

0,14 

0.06 

0.73 

0.21 

P 

0.03 

0.04 

0.93 

0.05 

0.10 

0.85 

0.08 

0.18 

0.73 

Lower  Level 


| - 

Probable  Error 

Sample  Error 

Test  Error 

P2 

P3 

P4 

P2 

P3 

P4 

P2 

P3 

P4 

P2 

0.91 

0.0 

0.09 

0.94 

0.04 

0.03 

0.87 

0.05 

0.08 

P3 

0.00 

1.0 

0.00 

0.04 

0.94 

0.01 

0.06 

0.92 

0.02 

P4 

0.01 

0.0 

0.99 

C.  16 

0.01 

0.83 

0.14 

0.01 

0.85 

TableXIV.  \ 2  Values 


Upper  L^vel 

Lower  Level 

Probable  Error 

Sample  Error 

27.90  reject 

12.13  accept 

39.61  reject 

5.43  accept 

The  value  of  X2  at  a  =  0.05  for  six  degrees  of  freedom  is  11.07.  The  re¬ 
sults  shown  in  Table  XIV  reject  the  hypothesis  that  the  probable  error  matrix 
is  a  good  predictor  of  performance,  and  accept  the  hypothesis  that  the  sample 
error  matrix  is  a  good  predictor  of  performance.  The  results  were  very  en¬ 
couraging  because  they  indicated  that  results  obtained  from  the  sample  set  were 
representative  of  those  to  be  expected  from  the  test  set.  In  previous  work,  re¬ 
sults  obtained  from  the  sample  set  were  usually  much  higher  than  those  from 
the  test  set. 

Although  the  probable  error  matrix  cannot  be  used  as  a  predictor  for  the 
sample  or  test  set,  it  could  have  considerable  value.  The  probable  error  matrix 
indicated  the  amount  of  overlap  caused  by  the  specific  set  of  discriminating  words 
used.  The  discrepancies  in  the  off-diagonal  elements  between  the  probable  error 
and  sample  error  matrices  were  due  to  the  dispersion  of  the  documents  in  the 
discriminant  space.  They  were  more  widely  dispersed  than  the  estimate  of  the 
dispersion  matrix  indicated. 
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Section  VI 


ANALYSIS  OF  CLASSIFICATION  EFFECTIVENESS 

The  effectiveness  of  the  classification  technique  and  the  causes  of  mis- 
classifications  are  analyzed  here  and  related  to  the  significant  classification 
parameters.  The  analysis  of  the  effect  of  variation  in  the  level  of  the  structure 
is  discussed  in  connection  with  each  parameter.  Some  new  parameters  which 
appear  to  be  adversely  affecting  classification  performance  are  identified. 
Classification  performance  is  reported  in  terms  of  several  efficiency  and 
effectiveness  measures.  Detailed  data  on  the  classification  parameters  and 
performance  are  recorded  in  the  tables  in  the  Appendix  and  are  discussed  in 
Section  V. 


NUMBER  OF  SAMPLE  DOCUMENTS  (EXPERIMENTS  1  AND  2) 

Classification  of  an  independent  test  set  of  documents,  as  well  as  the 
sample  documents  themselves,  was  performed  on  random  samples  of  35,  70 
and  140  documents  in  each  category  at  both  the  upper  and  lower  levels.  Two 
sets  of  discriminating  words  were  used,  one  at  each  level.  At  each  level  the 
set  of  discriminating  words  remained  fixed,  while  the  number  of  sample  docu¬ 
ments  was  changed.  Each  discriminating  word  set  contained  48  words;  16 
words  represented  each  category.  The  total  population  of  documents  at  the 
upper  level  was  1753  and  at  the  lower  level  was  872.  The  number  of  test  docu¬ 
ments  for  a  particular  run  was  the  total  population  minus  those  drawn  to  form 
the  sample.  Table  XV  shows  the  percentage  of  documents  classified  into  each 
category.  For  example,  88  per  cent  of  the  test  documents  manually  preclassi¬ 
fied  into  category  A  were  classified  automatically  into  category  A  based  on  a 
140-document  sample.  Five  per  cent  of  the  A  documents  were  misclassified 
into  M  and  seven  percent  were  misclassified  into  P. 

As  the  sample  size  increases ,  the  percentage  of  correct  test  documents 
increases  while  the  percentage  of  correct  sample  documents  decreases.  How¬ 
ever,  the  percentages  appear  to  be  approaching  a  common  point  within  each 
category  as  shown  in  Figure  4.  In  category  A  of  the  upper  level  the  test  per¬ 
centages  are:  0.79,  0.84  and  0.88,  while  the  sample  percentages  are:  0.97, 
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Table  XV.  Classification  Performance 


Upper  Level 


Sample  Documents 

Sample  Size 

35 

70 

140 

Effectiveness 

Actuai/Au:o 

A 

M 

P 

M 

P 

A 

M 

P 

A 

isa 

0.03 

0.0 

0.04 

0.88 

0.03 

0.09 

M 

ESS 

0.89 

0.09 

1 

0.81 

u.  04 

0.82 

0. 14 

P 

0.0 

0.06 

0.94 

'  v-' 

0.16 

0.78 

0.05 

0. 10 

0.85 

Test  Documents 

Sample  Size 

35 

70 

140 

Effectiveness 

■ 

Actual/Auto 

A 

M 

P 

| 

M 

P 

A 

M 

P 

A 

0.79 

0.05 

0.17 

0.84 

0.10 

0.07 

mm 

0.05 

0.07 

M 

0. 13 

0.64 

0.23 

0.09 

0.74 

0.17 

■m 

0.73 

0.21 

P 

0.11 

0.21 

0.68 

0.08 

0.30 

0.62 

0.08 

0.18 

0.73 

Lower  Level 


Sample  Documents 

35 

70 

140 

Effectiveness 

Actual/Auto 

P2 

P3 

P4 

P2 

P3 

P4 

P2 

P3 

P4 

P2 

0.94 

0.0 

0.0G 

0.96 

0.01 

0.03 

mm 

0.04 

0.03 

P3 

0.0 

1.0 

0.0 

0.04 

0.94 

0.01 

■Si 

0.94 

0.01 

P4 

0.06 

0.0 

0.94 

0.07 

0.01 

0.91 

0. 16 

0.01 

0.  83 1 

Test  Documents 

Sample  Size 

35 

70 

140 

Effectiveness 

Actual/Auto 

P2 

P3 

P4 

P2 

P3 

P4 

P2 

P3 

P4 

P2 

0.71 

mm 

mm 

0.  87 

0.05 

P3 

0.13 

0.06 

■B31 

P4 

0.26 

HH 

0.01 

0.83 

■9 

0.01 

0.85 
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Figure  4.  Effect  of  Classification  Performance  Dae  to  the  Number  of  Sample  Documents 


0,91  and  0.88.  In  category  P3  of  the  lower  level  the  test  percentages  are:  0.80, 
0.89  and  0.92  while  the  sample  percentages  are:  1.0,  0.94  and  0.94.  The  dis¬ 
persion  of  the  category's  mean  increases  as  sample  size  increases.  When  it  is 
small,  the  sample  documents  classify  well  but  are  not  representative  of  the 
whole  population;  hence  the  test  documents  do  not  classify  as  well.  As  the  dis¬ 
persion  increases,  the  sample  set  is  more  representative  of  the  population  and, 
even  though  the  sample  documents  perform  less  satisfactorily,  they  appear  to 
be  representing  the  population  more  faithfully.  An  increase  in  sample  size  be¬ 
yond  140  would  not  appear  to  offer  any  significant  increase  in  classification 
performance. 

Experiment  2  attempted  to  test  the  effect  on  classification  performance 
of  an  increase  in  the  number  of  reference  documents.  To  test  the  hypothesis, 
one  statistic  was  selected,  the  determinant  of  the  category's  dispersion.  It 
was  assumed  that  me  re  documents  would  be  misclassified  in  categories  having 
larger  dispersion.  Although  Table  XV  indicates  that  better  results  were 
obtained  with  an  increase  in  the  number  of  sample  documents,  the  hypothesis  * 
of  Experiment  2  must  be  rejected  on  the  basis  of  the  category's  dispersion. 
Apparently  the  magnitude  of  a  category’s  dispersion  is  not  a  good  indicator  of 
the  percentage  of  misclassifications  to  be  expected. 

The  number  of  sample  documents  also  affects  classification  performance 
through  the  selection  of  the  subset  of  discriminating  word  types,  and  Experi¬ 
ment  1  was  conducted  to  study  this.  Selected  words  have  a  high  discriminating 
coefficient  and  a  high  likelihood  of  occurring  in  many  documents .  If  words  are 
selected  from  too  small  a  sample,  they  may  not  have  a  high  discriminating 
coefficient  nor  a  high  likelihood  of  occurring  in  the  population.  Therefore, 
these  statistics  were  computed  on  samples  of  35,  70  and  140  documents.  Experi¬ 
ment  1  indicated  that,  according  to  the  test  used,  samples  of  35  and  70  were  too 
small  to  assure  reliable  estimates  of  the  discriminating  and  likelihood  statistics. 
However,  the  great  similarity  between  the  lists,  the  fact  that  the  rank  of  a  word 
on  the  list  of  selected  words  is  not  important,  and  the  general  effectiveness  of 
the  technique  even  when  based  on  a  35-  or  70-document  sample,  are  indications 
that  the  present  method  of  word  selection  is  adequate  until  further  experimen¬ 
tation  on  this  phase  of  the  technique  can  be  conducted. 


DOCUMENT  LENGTH  (EXPERIMENTS  3,  4,  AND  5) 

The  effect  of  document  length  on  classification  performance  was  analyzed 
in  terms  of  the  total  number  of  words  (tokens),  the  number  of  sentences,  and 
the  number  of  concepts  contained  in  each  document. 
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The  purpose  of  Experiment  3  was  to  determine  if  document  length  adversely 
affects  classification  performance.  The  results  of  the  experiment  indicated 
that  document  length  did  not  have  this  effect  since  no  significant  difference  was 
found  between  the  mean  lengths  of  correctly  and  incorrectly  classified  docu¬ 
ments.  In  data  bases  involving  a  greater  variation  in  document  length,  a  sig¬ 
nificant  difference  may  develop.  However,  in  data  bases  ouch  as  the  one  being 
tested,  where  the  variation  in  document  length  is  relatively  small,  it  did  not 
appear  to  be  a  significant  parameter.  Table  2  in  the  Appendix  contains  Informa¬ 
tion  on  classification  performance  relative  to  document  length.  A  slight  trend 
in  favor  of  longer  documents  exists  in  categories  M  and  P.  However,  there  are 
many  categories  in  which  short  documents  classified  as  well  as  long  documents. 
No  minimum  length  was  found  below  which  documents  were  consistently  mis- 
classified.  Experiment  3  also  revealed  that  no  significant  difference  existed 
in  the  mean  number  of  sentences  between  the  correctly  and  incorrectly  classi¬ 
fied  documents. 

Variations  in  other  parameters  related  to  document  length  may  have  more 
effect  than  the  document  length  itself.  These  include  the  number  of  sentences 
and  the  number  of  different  types,  hi  addition,  the  number  of  discriminating 
types  occurring  in  a  document  is  dependent  on  the  number  of  types  in  a  docu¬ 
ment.  Therefore,  an  additional  hypothesis  concerning  the  mean  number  of  dis¬ 
criminating  types  in  correctly  and  incorrectly  classified  documents  was  tested. 

A  significant  difference  existed  between  the  mean  number  of  discriminating 
types  (7. 6  and  5. 9)  occurring  in  the  correctly  and  incorrectly  classified  docu¬ 
ments  at  the  upper  level,  whereas  the  difference  between  the  means  (4.  S  and 
4. 3)  at  the  lower  level  was  found  to  be  insignificant,  hi  fact,  85  percent  of  the 
lower  level  documents  having  a  mean  of  4. 9  types  were  correctly  classified, 
whereas  only  76  percent  of  the  upper  level  documents  having  a  mean  of  7. 6 
types  were  correctly  classified.  A  definite  correlation  between  performance 
and  the  number  of  discriminating  types  can  be  observed  for  each  category  from 
Table  3  in  the  Appendix.  Classification  performance  would  be  increased  by 
increasing  the  number  of  types  occurring  in  each  document.  This  can  easily 
be  accomplished  by  increasing  the  number  of  discriminating  words.  The  per¬ 
formance  level  in  the  current  experiments  was  achieved  with  only  4S  discrimi¬ 
nating  types. 

Experimert  4  showed  that  the  cumulative  type -token  distribution  is  not 
dependent  on  document  length.  Hence,  one  can  infer  that  the  total  number  of 
words  occurring  in  documents  defining  a  category  is  more  important  than  the 
number  of  documents.  Similarly,  Experiment  5  indicated  that  the  number  of 
concepts  in  a  document  does  not  affect  the  number  of  types  occurring;  there¬ 
fore,  one  probably  needs  longer  multiple -concept  documents  to  achieve  the 
same  level  of  effectiveness  obtained  with  single-concept  documents . 
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HOMOGENEOUS  CATEGORIES  (EXPERIMENT  6) 


Lack  of  homogeneity  in  a  category  can  be  due  to  the  definition  of  the  subject 
categories,  the  sample  documents  and  the  discriminating  words  selected  to 
represent  the  categories.  Experiment  6  revealed  that  the  dispersion  of  a  cate¬ 
gory  Is  not  an  adequate  measure  of  its  homogeneity.  The  dispersion  of  an 
individual  category  neglects  the  amount  of  overlap  between  categories  and  the 
causes  of  this  overlap.  A  relation  between  the  overlap  of  categories  and  the 
discriminating  words  selected  to  represent  it  was  found  to  have  a  direct  effect 
on  classification  performance.  The  location  of  each  word  in  the  discriminant 
space  directly  influences  the  location  of  the  category. 

Figure  5  shows  a  plot  of  the  centers  for  categories  A,  M,  P  based  on  a 
140-document  sample.  The  "contours"^2)  for  each  category  are  the  locus  of 
points  within  which  95  per  cent  of  the  documents  belonging  to  that  category  are 
expected  to  lie.  Some  overlap  exists  which  indicates  an  ambiguous  region. 

The  coordinates  of  the  center  of  each  ellipse  are  functions  of  the  mean  frequen¬ 
cies  end  the  coefficients  of  the  discriminating  types .  The  coordinates  of  the 
categories'  centroids  and  dispersion  are  listed  in  Table  6  of  the  Appendix. 

Category  A  has  less  of  its  area  overlapped  by  other  categories,  and 
achieved  15  per  cent  more  correctly  classified  documents  than  either  of  the 
other  two.  Axis  I  provides  the  primary  discrimination  between  category  A  and 
categories  M  or  P.  Axis  n  provides  the  primary  discrimination  between  M  and 
P.  The  plot  of  tho  centers  indicates  the  change  required  to  increase  classifi¬ 
cation  periormance.  Categories  M  and  P  need  to  be  moved  farther  apart  along 
axis  II  and  need  to  be  farther  from  category  A  along  axis  I. 

The  relative  positions  of  the  ellipses  are  functions  of  the  sample  documents 
and  the  discriminating  words .  The  effect  of  each  word  on  the  position  of  the 
ellipse  can  be  observed  from  Figure  6,  The  coordinates  of  each  word  in  Figure 
6  are  obtained  by  multiplying  its  discriminant  coefficients  by  a  frequency  of 
one.  The  16  words  selected  to  represent  each  category  should  be  clustered 
near  their  category  mean.  The  words  SPIN  (0.10,  -  0.24)  and  CONDUC 
(0.07,  -  0.38)  are  examples  of  good  discriminators  for  category  P.  The  word 
REPORT  (0t17,  0.15)  is  an  example  of  a  poor  discriminator  for  category  P. 
The  word  REPORT  contributes  a  positive  increment  to  the  centroid  of  category 
P  causing  it  to  overlap  category  M.  The  word  TRANSI  (-0.10,  -C.13)  causes 
the  category  A  ellipse  to  overlap  category  P.  Words  having  a  high  coefficient 
which  initially  appear  to  be  good  discriminators  also  contribute  to  mis  classifi¬ 
cation.  DE-SCRI  (-0,27,  0.02)  and  DISCUS  (-0,37,  0.26)  are  examples  of  this 
type.  These  words  occurred  in  documents  belonging  to  all  three  categories 
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Category  M 


Centours"  of  Upper  Level  Categories,  140-Document  Sample 


Figure  6.  Location  of  Upper  Level  Words,  140-Document  Sample 


and  frequently  occuried  in  miselassified  documents  containing  three  types  or 
less.  The  classification  performance  of  documents  containing  five  word  types 
would  be  increased  by  the  deletion  of  this  type  of  word  from  the  discriminating 
word  set. 

Improvement  in  classification  performance  due  to  the  deletion  of  poor  dis¬ 
criminators  is  indicated  by  the  superior  performance  of  the  lower  level 
categories.  The  discriminating  word  set  for  the  lower  level  contained  only 
five  candidates  for  deletion,  whereas  the  upper  level  contained  14.  The  dis¬ 
criminant  coefficients  based  on  140-document  samples  are  shown  in  Table  XVI. 

Many  miselassified  documents  had  small  coordinate  values  which  placed 
them  near  the  origin  in  the  ambiguous  region.  Improving  the  homogeneity  of 
categories  should  reduce  the  amount  of  overlap ,  thereby  minimizing  the 
ambiguous  region  and  increasing  classification  performance. 


CLASSIFICATION  EFFECTIVENESS  MEASURES  (EXPERIMENT  8) 

It  would  be  desirable  to  use  the  probable  error  matrix  as  a  predictor  of 
classification  effectiveness,  since  then  it  would  be  unnecessary  to  classify  all 
the  sample  documents.  Experiment  8  revealed,  however,  that  the  sample 
error  matrix  is  a  better  effectiveness  measure.  This  result  is  no  doubt  due  to 
the  probable  error  matrix  indicating  the  error  to  be  expected  at  only  one  point, 
the  category  mean,  and  net  taking  the  dispersion  into  account,  whereas  the 
sample  error  matrix  does  since  it  is  based  or  many  documents. 

In  addition  to  these  measures  of  classification  effectiveness  based  on 
system  parameters,  it  is  desirable  to  consider  overall  performance  measures 
which  allow  comparison  of  this  technique  to  others .  Three  such  measures 
were  therefore  computed  for  each  category  at  both  the  upper  and  lower  levels. 
The  measures  computed  were  the  percentage  of  correctly  classified  documents , 
Swets '  rr.tios  and  Cleverdon's  ratios ^). 

Swets’  and  Cleverdon’s  ratios  are  based  on  the  familiar  two-by-two 
contingency  table.  (See  Table  XVII .}  Swets  uses  the  ratios  to  compute  an 
additional  performance  measure,  E,  v’hich  is  the  statistical  operating  charac¬ 
teristic  capable  of  showing  the  amount  of  type  I  and  n  errors.  Values  of  E  can 
be  interpreted  objectively  since  it  has  been  widely  used  in  other  statistical 
applications.  Curves  of  the  operating  statistic  are  available  for  the  complete 
range  of  relevancy  from  0.0  to  1.0  (Reference  13).  The  operating  characteris¬ 
tic,  in  addition  to  providing  a  measure  of  effectiveness,  also  provides  a 
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Table  XVl(a).  Normalized  Discriminant  Coefficients  for  Upper  Level  Categories, 

140-Document  Sample 


I 

n 

I 

n 

cmcui 

-0. 17 

0.  C2 

DIFFUS 

0.  09 

0, 12 

DESCRI 

-0.20 

0. 19 

STRUCT 

0.  09 

0. 17 

OPERAT 

-0.32 

-0.05 

SINGLE 

0. 11 

-0.  01 

OUTPUT 

-0. 03 

0.  07 

CONCEN 

-0.  03 

0. 13 

TRANSI 

-0. 10 

-0. 13 

C 

0.09 

0. 14 

DEVICE 

-0.41 

0. 10 

CONTAI 

0.08 

0.  07 

DESIGN 

-•0.36 

-0. 05 

OXYGEN 

0.06 

0. 18 

PROVID 

-0. 15 

0.  01 

TECHNI 

0.  07 

0.26 

SIGNAL 

-0.25 

0.01 

FIELD 

0.  01 

-0. 16 

SWITCH 

-0. 19 

-0.  02 

K 

0.03 

-0.  14 

VOLTAG 

-0. 19 

-0.  06 

CONDUC 

0.  07 

-0. 38 

OSCILL 

-0.32 

0.09 

EXPER) 

0.08 

-0. 17 

CONTRO 

0.04 

0. 10 

MAGNET 

-0.  05 

-0. 12 

AMPUF 

-0.04 

-0. 02 

RESONA 

0.  00 

-0.29 

PULSE 

-0. 14 

0. 00 

SPIN 

0. 10 

-0.24 

SYSTEM 

-0. 12 

0. 17 

ELECTR 

0.  02 

-0.06 

CRYSTA 

0.  09 

0.  09 

MEASUR 

-0.  02 

-0. 12 

GROWTH 

0. 13 

0.  22 

EFFECT 

0.04 

-0. 17 

DISLOC 

0.  05 

0. 13 

RESULT 

0. 18 

-0.06 

SURFAC 

0.  03 

0.  03 

REPORT 

0. 17 

0. 15 

METHOD 

0. 14 

0. 14 

DEPEND 

0.  04 

-0. 15 

FOUND 

0.06 

0. 12 

OBSERV 

0.  05 

-0.06 

IMPURI 

0. 13 

0.  21 

TEMPER 

-0.03 

-0.  07 

DISCUS 

0. 05 

0. 13 

BETWEE 

0. 10 

-0.  20 
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Table  XVI(b).  Normalized  Discriminant  Coefficients  for  Lower  Level  Categories, 

140-Document  Sample 


I 

n 

I 

II 

SUPERC 

0. 14 

-0.39 

IRON 

-0. 33 

0.  15 

RESIST 

0.01 

-0.07 

INTERA 

-0. 10 

0.  03 

SEMICO 

0.  14 

-0.23 

ANTIFE 

0.  02 

0.  'M 

HALL 

0. 13 

-0. 19 

ANISOT 

-0.  07 

0.  07 

ELECTR 

0.05 

-0.  06 

RELAXA 

-0. 17 

0.  08 

MOBILI 

-0.01 

-0.07 

SUSCEP 

-0.  21 

0.  14 

CARRIE 

0.  06 

-0.  04 

IONS 

-0.  06 

-0.  02 

CONDUC 

0.  03 

-0. 14 

MOMENT 

0.  08 

-0.  02 

CURREN 

0.01 

-0.  05 

ABSORP 

0. 12 

0. 17 

SURFAC 

0.  05 

-0. 10 

PHOSPH 

0. 12 

0.  26 

DENSIT 

-0.  04 

-0.  20 

EMISSI 

0.  18 

0.  27 

RECOMB 

0.  06 

-0.  20 

LIGHT 

0.  02 

0. 17 

TYPE 

0. 10 

-0.  06 

LUMINE 

0.11 

0.  17 

FIELD 

0.  00 

-0.  03 

BAND 

0.  01 

0.  04 

SCATTE 

-0.  05 

-0. 10 

OPTICA 

0.  09 

0.  16 

METHOD 

-0.03 

-0. 13 

EXCITA 

0.08 

0.  11 

MAGNET 

-0.23 

0.03 

WAVELE 

0.  26 

0.  37 

SPIN 

-0. 15 

0.  01 

EDGE 

0.  01 

0.  01 

RESONA 

-0.  22 

-0. 04 

BANDS 

0.02 

-0.  01 

FERROM 

-0.  29 

0. 15 

RADIAT 

0.  15 

0. 19 

SATURA 

-0.  21 

0.00 

MICRON 

0.  09 

0.  18 

PARAMA 

-0.28 

-0. 01 

INTENS 

-0.  03 

0.14 

EXCHAN 

-0.  29 

0. 07 

FLUORE 

0. 10 

0.  07 

FERRIT 

-0.32 

0. 09 

SPECTR 

-0.  02 

0.  10 
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measure  of  the  breadth  of  a  particular  query.  In  an  ideal  system  E  would  be 
constant,  and  the  breadth  of  a  query  could  be  determined  by  the  slope  of  the 
curve  at  the  point  resulting  from  the  relevance  values  of  that  query. 


Table  XVII.  Pertinency  and  Retrieval  2x2  Contingency  Table 


Retrieved 

Pertinent 

Not  Pertinent 

Total 

a 

b 

a  +  b 

Not  Retrieved 

c 

d 

c  +  d 

Total 

a  +  c 

b  +  d 

a  +  b  +  c  +  d 

Swets  calls  his  ratios  the  conditional  probability  of  a  hit,  Prp(R)  =  a/a+c; 
and  the  conditional  probability  of  a  false  drop,  Prjj  (R)  =  b/b+d.  The  condi¬ 
tional  probability  of  a  hit  becomes  the  ordinate  of  the  operating  characteristic 
and  the  conditional  probability  of  a  false  drop  becomes  its  abscissa.  Figure  7 
shows  a  family  of  normal  operating  characteristic  curves  plotted  on  a  double 
probability  scale.  A  value  of  E  =  0  indicates  a  random  system  regardless  of 
the  percentage  6f  hits.  A  hit  ratio  of  0.70  and  a  false  drop  ratio  of  0.70  would 
yield  an  E  of  zero.  The  greatest  value  of  the  operating  characteristic  occurs 
in  the  analysis  of  misclassification  errors  and  the  design  of  systems.  It  indi¬ 
cates  the  direction  the  analysis  should  proceed  to  improve  system  performance. 
The  values  of  a  system  on  the  curve  will  indicate  whether  a  greater  increase  in 
performance  can  be  expected  by  increasing  the  hit  ratio  or  decreasing  the  false 
drop  ratio.  Points  of  diminishing  return  can  also  be  found  from  the  curves. 
Cleverdon  calls  his  ratios  recall,  a/a+c;  relevance,  a/a+b;  pnd  precision, 
a+b/ a+c.  These  ratios  have  limited  value  in  analyzing  effectiveness  since 
their  theoretical  distributions  are  not  ’*^own  and  the  size  of  the  document 
collection  is  ignored.  The  element  d  of  the  two-by-two  table  does  not  enter  into 
any  of  the  computations.  Values  of  Cleverdon' s  ratios  are  reported  in  Table 
5  of  the  Appendix  for  comparisons  with  other  retrieval  techniques. 

Classification  effectiveness  values  of  the  upper  and  lower  level  categories 
are  plotted  in  Figure  7(a).  The  clustering  of  the  lower  level  categories  indi¬ 
cates  a  fairly  consistent  system  performance.  The  effect  of  a  change  in  sample 
size  is  shown  in  Figure  7(b).  Increasing  the  sample  size  raised  the  performance 
level  fay  increasing  the  hit  ratio  end  decreasing  the  false  drop  ratio.  Since  all 
plots  are  to  the  left  of  a  diagonal  perpendicular  to  E  =  0  line,  the  point  of 
diminishing  returns  has  not  been  reached. 
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Section  VII 


RECOMMENDATIONS 


Experiments  on  the  principal  classification  parameters  have  produced 
valuable  information  concerning  their  effect  on  classification  performance. 

Also,  new  parameters  have  been  identified  which  appear  to  have  an  important 
relationship  to  performance.  This  section  discusses  the  most  significant  of 
these  and  recommends  specific  areas  for  further  study  and  experimentation. 

Since  the  specific  word  types  retained  in  the  discriminating  word  set 
directly  affect  the  location  and  size  of  the  category  ellipses,  experiments 
should  be  performed  involving  the  deletion  of  certain  word  types  from  the  set 
based  on  classification  performance.  The  word  selection  criteria  should  be 
investigated  and  the  possible  inclusion  of  the  multivariate  discriminant  co¬ 
efficient  in  the  criteria  should  be  examined. 

The  decision  rule  employed  in  the  current  experiments  was  based  on  the 
X2  value  of  each  document.  This  did  not  consider  the  a  priori  probability 
distribution  of  the  documents.  For  example,  it  was  known  that  there  were 
approximately  four  times  as  many  P  documents  as  A  and  M  documents  in  the 
test  population.  Use  of  a  probability  measure  weighted  according  to  the  a 
priori  probability  of  each  category  may  offer  a  significant  improvement.  An 
expression  for  a  probability  measure  is  given  by  equation  (14). 

Since  there  will  always  be  a  region  of  ambiguity  in  the  area  of  overlapping 
categories  near  the  origin,  documents  lying  within  this  region  should  not  be 
classified.  A  criterion  such  as  minimum  number  of  word  types  or  minimum 
radius  should  be  investigated  to  identify  such  documents.  The  use  of  such  a 
criterion  would  also  yield  information  concerning  the  efficiency  of  the  technique. 
The  performance  level  of  the  technique  on  the  set  of  documents  meeting  the  cri¬ 
terion  may  be  sufficiently  high  to  warrant  operational  testing.  If  90  percent  of 
a  document  corpus  could  be  classified  automatically  with  95  percent  effective¬ 
ness,  and  the  remaining  10  percent  could  be  identified  automatically,  then  it 
would  be  a  minor  problem  to  classify  these  manually. 

A  sufficient  number  of  the  significant  classification  parameters  have 
been  identified  and  studied  to  warrant  testing  the  classification  technique  on 
documents  selected  from  an  operational  environment.  Such  testing  should  be 
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directed  toward  identifying  any  remaining  problems  in  the  application  of  the 
technique,  as  well  as  toward  the  further  improvement  of  classification  effi¬ 
ciency  and  effectiveness. 


Section  V  III 


SUMMARY 


Classification  experiments  were  performed  on  a  d-ta  base  of  approxi¬ 
mately  2700  abstracts  from  the  solid  state  physics  fiel* The  purpose  of 
these  experiments  was  to  assess  the  effect  of  certain  classification  para¬ 
meters  on  the  effectiveness  and  utility  of  a  document  classification  technique 
based  upon  multiple  discriminant  functions. 

Multiple  discriminant  functions  derived  from  word  frequency  information 
in  manually  classified  documents  are  used  to  test  whether  or  not  an  incoming 
document  should  be  assigned  to  a  particular  category.  Since  a  test  is  made 
for  each  category,  documents  could  be  assigned  to  more  than  one  category,  and 
any  number  of  categories  can  be  handled  by  the  technique.  In  applications  re¬ 
quiring  subcategories  or  a  hierarchical  structure,  the  teennique  would  classify 
documents  into  the  lowest  level  by  making  sequential  decisions  at  each  level. 

Of  all  the  parameters  investigated,  the  number  of  sample  documents 
used  to  define  a  category  yielded  the  greatest  increase  in  classification  per¬ 
formance.  Experiments  were  conducted  with  three  different  sample  sizes: 

35,  70,  and  140  documents  in  each  category  for  a  total  of  105,  210  and  420 
documents. 

The  percentage  of  correctly  classified  test  documents  increased  from 
0, 69  to  0. 76  at  the  upper  level  and  from  0. 74  to  0.  88  at  the  lower  level.  An 
interesting  effect  of  the  opposite  nature  with  respect  to  the  sample  documents 
wa3  observed.  As  the  number  of  sample  documents  was  increased,  the  per¬ 
centage  of  correctly  classified  sample  documents  decreased  from  0.  93  to 
0.  85  at  the  upper  level  and  from  0.  96  to  0.  90  at  the  lower  level.  Although 
better  results  were  obtained  in  the  sample  set  using  a  smaller  number  of 
documents,  the  estimate  of  the  coefficients  was  apparently  not  representa¬ 
tive  enough  of  the  whole  population.  As  the  number  of  sample  documents  was 
increased,  the  percentage  of  correctly  classified  sample  and  test  documents 
approached  a  common  point,  indicating  that  the  sample  is  most  representative 
of  the  population. 

The  number  of  sample  documents  also  affects  classification  performance 
through  the  subset  of  word  types  selected  to  represent  the  category,  Since 


each  word  does  not  occur  in  every  document,  its  occurrence  is  dependent  on 
the  number  of  documents  samp  3d.  Selection  decisions  based  on  samples  that 
are  too  small  may  omit  words  that  are  truly  good  discriminators.  List  of 
words  were  ranked  according  to  their  univariate  discriminant  coefficients 
based  on  various  sample  sizes,  and  a  comparison  of  the  first  50  words  on 
each  list  were  made.  A  rank  correlation  test  revealed  that  the  subset  of 
words  from  the  35-  and  70-document  samples  were  significantly  different 
from  the  140-document  sample.  Therefore,  when  selecting  subsets  of  words, 
descriptors  or  keywords,  one  must  be  assured  that  the  sample  size  Is  large 
enough  to  adequately  represent  the  sample. 

The  variation  in  document  length  did  not  adversely  affect  classification 
performance.  The  range  of  the  length  of  the  abstracts  was  approximately  20 
to  250  tokens.  Tests  for  a  significant  difference  between  the  mean  document 
lengths  and  the  mean  numbers  of  sentences  in  the  correctly  and  incorrectly 
classified  documents  revealed  no  significant  differences. 

A  more  significant  effect  or:  classification  performance  was  due  to  the 
number  of  discriminating  word  types  occurring  in  a  document.  Although 
documents  containing  12  or  more  of  the  48  possible  types  were  usually  classi¬ 
fied  correctly,  no  minimum  below  which  documents  were  usually  misclassified 
was  found.  Misclassification  was  more  often  due  to  the  specific  discriminating 
words  found  in  the  document,  rather  than  merely  the  number.  The  contribu¬ 
tion  of  specific  word  types  to  a  category’ s  mean  and  dispersion,  and  conse¬ 
quent  adverse  effect  on  classification  performance,  was  identified  and  war¬ 
rants  further  investigation. 

The  arrival  rate  of  new  word  types  was  found  to  be  approaching  a  reason-  „ 
ably  small  number  after  several  hundred  documents  had  been  counted.  This 
assured  that  a  subset  of  discriminating  words  can  be  selected  on  the  basis  of 
an  economical  number  of  tokens  represented  by  a  small  number  of  sample  docu¬ 
ments. 

Considerable  variation  in  performance  was  found  among  categories.  The 
primary  variation  was  found  to  be  due  to  the  words  selected  to  represent  the 
categories.  Those  categories  containing  words  representing  more  than  one 
category  did  not  perform  as  well  as  other  categories.  An  increase  in  per¬ 
formance  in  those  categories  could  be  achieved  by  better  selection  of  the 
words  representing  them,  thereby  increasing  overall  performance. 

The  ability  of  the  technique  to  effectively  classify  at  more  than  one  level 
was  demonstrated  by  the  results  achieved  in  several  experiments. 
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investigation  of  effectiveness  measures  revealed  that  Swets1  E  appeared 
to  be  the  best  statistic  to  measure  system  performance  and  to  analyze  the 
causes  of  misclassifications. 

Experimentation  has  indicated  that  classification  effectiveness  as  high 
as  02  percent  can  be  expected.  Classification  parameters  have  been  identified 
vhich  will  aid  in  the  analysis  of  causes  of  misoiassified  documents,  and  pro* 
vide  guidance  in  the  application  of  this  classification  technique  to  new  data 
bases.  Sufficiently  high  performance  levels  have  been  achieved  to  warrant 
experimental  application  of  the  multiple  discriminant  technique  to  operational 
data  bases  in  order  to  identify  any  remaining  problems,  and  to  further  im¬ 
prove  classification  efficiency  and  effectiveness. 
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Appendix 

DETAILED  DATA  ON  CLASSIFICATION 
PARAMETERS  AND  PERFORMANCE 
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CATEGORY  P4  PERTINENT  NOT  PERTINENT  RATIO  RATIO  RATIO 

RETRIEVED  0.67  0.11  0.67  0.51  1.32 
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MEAN  S.D. 

CORRECT  OOCUMENTS  5.45  2.34 

INCORRECT  DOCUMENTS  4.B3  2.37 
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Table  7.  Sample  Error  and  X  Among  Category  Centroids 


a.  Upper  Level 


35 

70 

140 

A 

M 

P 

A 

M 

P 

A 

M 

P 

Sample 

1.0 

0.0 

1.0 

0.0 

0.0 

1 

0.0 

Error 

0.0 

0.99 

0.02 

0.83 

0.15 

I 

0.93 

0.06 

0.0 

0.0 

0.01 

0.14 

0.85 

'  '  1 

0.04 

0.93 

0.0 

24.1 

25.0 

0.0 

16.1 

14.7 

0.0 

18.8 

13.4 

x2 

17.4 

0.0 

10.4 

8.1 

0.0 

3.1 

9.3 

0.0 

4.8 

22.2 

12.2 

0.0 

11.4 

3.9 

0.0 

6.6 

7.1 

b.  Lower  Level 


35 

70 

140 

P2 

P4 

P2 

P3 

P4 

P2 

P3 

P4 

Sample 

1.0 

0.0 

0.0 

0. 99 

0.0 

0.01 

0.91 

0.0 

0.09 

Error 

0.0 

1.0 

0.0 

0.0 

1.0 

0.0 

0.0 

1.0 

0.0 

0.0 

0.0 

1.0 

0.0 

0.0 

1.0 

0.01 

0.0 

0.99 

o 

0.0 

24.9 

10.6 

0.0 

16.8 

8.3 

ill 

10.1 

4.8 

X 

33.0 

0.0 

56.0 

28.6 

0.0 

23.7 

0.0 

32.4 

16.1 

23.7 

0.0 

15.5 

23.1 

0.0 

Hi 

13.3 

0.0 
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